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Executive Summary 


The overall purpose of the research studies described in this report was to investigate the 
quality of the administration of the Partnership for Assessment of Readiness for College and 
Careers (PARCC) assessment during its first operational year (2014 — 2015). Findings from these 
studies can be used to identify threats to the validity of PARCC test administration so that those 
threats can be addressed and reconciled for future test administrations. 


The research studies in this report represent an extension of the Quality of Test Administration 
Study from the PARCC field test (Sinclair, Deatz, Johnston-Fisher, Levinson, & Thacker, 2015). 
The guiding framework for the current Quality of Test Administration Study and the prior 
Quality of Test Administration Study (field test) comes from the Theory of Action (TOA) in the 
PARCC validity studies memorandum (Thacker, Sinclair, Wise, & Becker, 2014). The claims from 
the Test Administration phase of the TOA are listed in Figure 1. These are the claims being 
investigated in the current study. 


va Nolaallalicvee-liceyal 


* Claim 1: Administrators 
and teachers* are prepared 
to administer the 
assessments as intended. 


* Claim 2: Teachers* have 
the resources and supports 
to administer the 
assessments. 


° Claim 3: Technology 
improves and facilitates the 
assessment experience. 


* Claim 4: Students respond 
to items as intended. 


Figure 1. Claims from the Administration Phase of the PARCC Theory of Action. 


*The Quality of Test Administration research studies focus specifically on Test Administrators (TAs). Consequently, 
for the purposes of this report, Claims 1 and 2 are investigated specifically for TAs. 
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The findings from the research studies on test administration provide evidence in relation to 
the claims from the Test Administration phase of the PARCC TOA. A summary of these findings 
is presented next. Throughout this report, references are made to activities that address the 
Standards for Educational and Psychological Testing (hereafter referred to as the Standards 
[AERA, APA, & NCME, 2014)]). 


Claim 1: TAs are Prepared to Administer the Assessments as Intended 


In order for this claim to be true important assumptions that must be verified are: (a) TAs are 
trained to administer the assessments and (b) training is effective. Findings related to several of 
the research questions provide evidence for these assumptions and for the overall claim of 
preparedness. 


The research questions that provide evidence related to Claim 1 are: 


e Research Question 2: To what degree do the TAs follow the protocols and instructions? 
(addresses Standards 6.1 and 6.2) 

e Research Question 6: Are there any apparent attempts to record or copy test materials 
including the test questions by students or others? (addresses Standards 6.6 and 6.7) 

e Research Question 8: If any disruptions, interruptions, or other problems occurred, did 
the test administrators deal with the issue appropriately and effectively? (addresses 
Standards 6.1 and 6.3) 

o Research Question 8a: Were the TAs’ actions informed by training, by background 
experience, or a combination of the two? (addresses Standard 6.1) 
o Research Question 8b: Would a different approach have been more effective? 

e Research Question 9: Was security of test materials maintained at all times? (addresses 

Standards 6.6 and 6.7) 


The following support was evidenced for Claim 1 and for the research questions mapped to 
Claim 1: 


e The majority of TAs administering CBTs and PBTs reported that the PARCC online 
trainings, particularly the Introduction to PARCC module, effectively prepared them to 
administer the PARCC assessments. 

e Even higher percentages of TAs reported that the training they received from their 
school effectively prepared them to administer PARCC assessments. 

e Most TAs reported that they gained additional familiarity with PARCC content via 
sample items, practice tests, and the PARCC tutorial on TestNav (CBT only) prior to test 
administration. 
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e TAs followed standardized procedures during test administration with very few 
exceptions, and test security was maintained at all times with no apparent attempts to 
record or copy test materials. 

e In only one of the 109 observed test sessions did the TA deviate from the test 
administration script; this represented a notable improvement from the field test. 

e The vast majority of TAs reported reading the Accessibility Features and 
Accommodations (AFA) Manual prior to test administration; this was a notable increase 
from the field test. 

e The majority of TAs agreed that the AFA trainings and materials effectively prepared 
them to administer AFAs. 


Potential areas on which to focus/take into consideration: 


e Just over half of the TAs indicated that the PARCC trainings/materials effectively 
prepared them to handle basic technology-related problems (CBT). While still a 
majority, this result compares less favorably than other findings regarding the 
effectiveness of trainings/materials. 

e Only about one-third of TAs administering PBTs indicated that they worked with the 
PARCC paper-based student tutorials prior to test administration. 

e Most TAs were unaware of whether the AFAs that were pre-identified in the PNP were 
made available to students during a practice session, and most were also unaware as to 
whether changes were made to a student’s PNP or AFAs during test administration. 


Suggestions for consideration to help further strengthen support for Claim 1: 


e Ensure that schools are well-informed in all aspects of PARCC test administration well in 
advance of the testing windows. The type of training that received the highest 
effectiveness rating was the training provided at the school-level. This underscores the 
importance of ensuring that schools have all the resources and materials they need to 
effectively train their staff well in advance of the testing window. This includes ensuring 
that the finalized TA Manuals are available in a timely manner. 

e More emphasis should be placed on ensuring that all TAs complete the PARCC tutorials 
prior to test administration, particularly for TAs administering PBTs. Of those who 
completed the tutorials, the vast majority agreed that the tutorials helped better 
prepare them to administer the PARCC assessments. Nonetheless, approximately one- 
third of TAs administering CBTs indicated that they did not review the PARCC tutorials 
prior to test administration and two-thirds of TAs administering PBTs indicated that they 
did not review the PARCC tutorials prior to test administration. (/t should be noted that 
PARCC has placed additional emphasis on the tutorials and practice test sections in the 
manual to further emphasize the importance of administering these to students prior to 
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testing. PARCC has also created a pre-administration guidance document that has a 
whole section devoted to instructional supports and to encourage the use of tutorials 
and practice tests.) 

e Enhance the training on handling basic technology-related problems. Only slightly more 
than half of the TAs agreed that the PARCC online trainings effectively prepared them to 
resolve basic problems related to technology during test administration. Particular 
attention should be given to handling local technology-related issues with logging 
students onto the assessment and with students getting kicked off the assessment as 
these were the two most frequently reported types of technology-related problems 
encountered during test administration. (/t should be noted that PARCC has changed the 
PearsonAccess"*t module to be broken into smaller, task-based modules. The aim of this 
change is that TAs will be able to view the shorter modules just prior to testing to refresh 
their memory on basic technology functions for testing. PARCC also added a new 
troubleshooting computer-based errors section to the manuals with direct links to 
common error messages to help provide transparency and a direct link to support 
information.) 

e Provide additional guidance to TAs on how to handle student questions that arise during 
testing. In particular, provide guidance on how to handle student questions on 
navigating through the test. 

e Rethink the guidance provided in the TA Manual for the use of the timing box during 
test administration. Many TAs did not use the timing box exactly as it is described in the 
TA Manual. For example, many TAs provided verbal updates on time remaining as 
opposed to writing the time remaining in a timing box. Also, many TAs provided more 
frequent updates on time remaining than what is specified in the TA Manual. If flexibility 
in managing time is acceptable, consider revising the TA Manual to indicate that the 
timing box is one example of how to manage the time during the test session. 
Otherwise, if flexibility in managing time is not desirable, then consider providing more 
definitive instructions in the TA Manual on how time must be managed via use of a 
timing box. (/t should be noted that PARCC has added an improved materials/timing box 
to the script to add clarity for tracking time during testing.) 

e Improve the process for identifying AFAs in advance via the Personal Needs Profile (PNP) 
so that it is clearer and easier to follow. Also, the majority of TAs responded either “no” 
or “not sure” when asked whether all the AFAs identified in the PNP were made 
available to students during a practice session. Consequently, PARCC may want to 
incorporate additional scrutiny to ensure that students who need AFAs are indeed given 
the opportunity to practice with those AFAs prior to actual test administration. (/t 
should be noted that PARCC has changed the PNP process to be identified during the 
student registration process so that this information is gathered earlier. Additionally, the 


Submitted September 28, 2015 Vv 


PARCC Quality of Test Administration 


PARCC Accessibility Features and Accommodations Manual has been revised to include a 
whole section devoted to the tasks to be completed before, during, and after testing for 
each accommodation. This revision more clearly outlines the tasks that need to be 
completed related to accommodations. PARCC has also placed additional emphasis on 
the tutorials and practice test sections in the manual to further emphasize the 
importance of administering these to students prior to testing. PARCC also created a pre- 
administration guidance document that has a whole section devoted to instructional 
supports and to encourage the use of tutorials and practice tests.) 


Claim 2: TAs Have the Resources and Supports to Administer the Assessments 


For Claim 2 to be true important assumptions that must be met are: (a) resources and supports 
must be clear, sufficiently detailed, and easy to follow, and (b) resources and supports must not 
be overly burdensome for TAs to use and apply. 


The research questions that provide evidence for this claim and these assumptions are: 


e Research Question 1: To what degree do the test administrators (TAs) find the 
instructions clear, sufficiently detailed, and easy to follow? (addresses Standard 4.15) 

e Research Question 10: Did the test administration create minimal disruption to the 
school and staff? 


The following support was evidenced for Claim 2 and for the research questions mapped to 
Claim 2: 


e The vast majority of TAs for Year 1 agreed that the policies and procedures in the TA 
Manual were easy for them to understand, a notable increase from the field test. 

e The vast majority of TAs for Year 1 agreed that the instructions (including the scripts) in 
the TA Manual were easy for them to implement, a notable increase from the field test. 


Potential areas on which to focus/take into consideration: 


e Most TAs and TCs stated that the administration of the PARCC assessment required 
more time and resources to administer than previous state tests. 

e The TAs and TCs indicated that the number of staff hours required for preparation was 
greater for the administration of the PARCC assessment than for prior state tests. 

e The TAs and TCs indicated that PARCC was more disruptive to the curriculum schedule 
than prior state tests. 


The primary recommendation to help strengthen support for Claim 2 is one that is already 
being implemented by PARCC—that is, to reduce the amount of resources necessary to 
administer PARCC by collapsing the PBA and EOY testing windows. 
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Claim 3: Technology Improves and Facilitates the Assessment Experience 


For this claim to be true an important assumption that must be met is that the technology must 


work as intended. If the technology does not work as intended—for example, if there are 
functionality problems and/or delays in its application, then the veracity of this claim is 


threatened. 
The evidence for this claim comes directly from Research Question 7a: 


e Are there any technology-related problems during test administration? (addresses 
Standards 6.3 and 6.4) 


The following support was evidenced for Claim 3 and for the research question mapped to 
Claim 3: 


e Fewer problems with technology were observed during test administration in Year 1 and 
when problems did occur, the severity of the problems was much less than those 


observed during the field test. 


e Students reported substantially fewer problems with the technology in Year 1 than in 


the field test. 


e Very few students reported having problems with the highlighter tool, the magnification 
tool, the calculator tool (mathematics) and navigating between passages/stories (ELA), 


which was consistent with findings from the field test. 


Potential areas on which to focus/take into consideration: 


e Most TAs reported that they encountered one or more problems related to technology 


during the test administration. These findings were similar to the findings from the field 


test. (/t is important to note that PARCC has added a section to the manuals to provide 
additional emphasis and guidance on the importance of conducting an infrastructure 


trial to reduce the frequency of technology related problems.) 


e TAs most frequently encountered problems with logging into the assessment and with 


the system disconnecting/logging off. 

e Students also reported “difficulty logging on” as one of the most frequently 
encountered technology-related problems. 

e Students also reported difficulties with dragging or moving objects on the screen. 

e Some students reported difficulty entering math symbols and numbers. 

e Some students reported difficulty finding information in stories/passages. 

e There are substantive numbers of students who are not using the tools/features 
embedded in the computer-based tests (CBTs). 
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Suggestions for consideration to help further strengthen support for Claim 3: 


e Provide a more targeted focus on reconciling local technology-related issues with 
getting students logged on and with the system disconnecting. 

e Place more emphasis on ensuring that students take advantage of the tools embedded 
within the CBTs given that substantive numbers of students indicated that they did not 
use these tools and features. 

e Provide students with additional practice on entering math symbols and numbers prior 
to test administration. (/t should be noted that PARCC has developed a standalone 
equation editor for teachers to use with students for advanced practice.) 

e Provide students with additional practice on finding information in passages/stories on 
the ELA assessment prior to test administration. 


Claim 4: Students Respond to Items as Intended 


Finally, for Claim 4 to be true, several assumptions must be verified. First, for students to 
respond to items as intended, they must understand the directions read by the test 
administrator (i.e., the script). Second, students must also understand the directions they read 
for the questions on the test. If the students do not understand the directions read by the TA 
and/or if they are confused by the directions they read on the test, then they may not respond 
to the items as intended. Third, to respond to items as intended, students need to have some 
engagement with the assessment. If they are not engaged or if they have a low level of 
engagement, then they might not sufficiently attend to items so as to respond to them as 
intended. Many factors might negatively impact student engagement. 


The research questions that provide evidence for this claim and these assumptions are: 


e Research Question 3: Do the students appear to understand the instructions provided to 
them? (addresses Standards 4.16 and 6.5) 
e Research Question 4: To what degree are students engaged in taking the test? 
(addresses Standards 4.16, 6.3, 6.4, and 6.5) 
e Research Question 5: Is there any disruptive student behavior during the session? 
(addresses Standard 6.4) 
e Research Question 7: Are there any interruptions during the session? (addresses 
Standards 6.3 and 6.4) 
o Research Question 7a: Are there any technology-related problems during test 
administration? 
o Research Question 7b: What type of questions, if any, do students ask during the 
test administration? (also relevant to Standards 4.16 and 6.5) 


Submitted September 28, 2015 viii 


PARCC Quality of Test Administration 


The following support was evidenced for Claim 4 and for the research questions mapped to 
Claim 4: 


e The vast majority of students report that they understood the directions (i.e., script) 
read by their TA. This finding was similar to the finding from the field test. 

e The vast majority of TAs agreed that students appeared to understand the instructions 
they read to them and that the instructions covered all of the information necessary to 
take the test. These findings reflect a notable increase from the field test to Year 1, 
which suggests that the improvements that PARCC made to the script in the TA Manual 
since the field test were beneficial. 

e TAs effectively kept distractions such as, outside visitors and disruptive student 
behaviors to a minimum. 

e Technology-related problems were not likely to be a major disruption, as they were in 
the field test, given that most issues were minor in nature and resolved within a couple 
minutes. 

e In nearly all instances, the TAs handled student questions appropriately, thereby 
suggesting that distractions caused by students asking questions were minimized. 

e Most of the questions on the test covered topics students had already learned about in 
school. 

e Most students were very familiar with the mode of assessment (CBT) and they preferred 
that mode of assessment. 

e Feedback obtained from the TAs in the debriefing interviews and comments made by 
observers on the observation checklist indicated moderate to high levels of student 
engagement. 


Potential areas on which to focus/take into consideration: 


e The Student Surveys asked students, “How often was it hard to understand the 
directions for the questions on this test?” the majority of students responded that it 
was hard to understand the directions for the questions on the test “some of the 
time” and nearly a third of students taking the mathematics assessments and nearly 
a fourth of students taking the ELA assessments responded that it was hard to 
understand the directions for the questions on the test “most or almost all of the 
time.” These findings were similar to the findings from the field test. 

e Test difficulty emerged as a potential factor that might have negatively impacted 
student engagement, particularly for the mathematics assessments. Most students 
reported that the mathematics assessments were harder than their school work. 
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Suggestions for consideration to help further strengthen support for Claim 4: 


e Improve directions for questions on the test, such as simplifying the instructions for 
answering the questions, to enhance students’ understanding. Roughly one-third of TAs 
indicated that students asked questions about where to mark their answers and enter 
their responses (CBT and PBT), and more than one-third indicated that students asked 
questions about how to find and use tools (CBT). Consequently, these might be two 
aspects of instructions that could be clarified. If students are better able to understand 
the instructions on the test, then the test might not seem so difficult to students. 

e Encourage greater participation in opportunities to practice with PARCC content prior to 
test administration, particularly completion of the student tutorials, as results indicate 
that fewer students practiced with tutorials than with sample items and practice tests. 
Greater familiarity to the PARCC content prior to test administration may also help 
students better understand the instructions on the test. 


Conclusion 


Overall, the findings from the Test Administration Study for Year 1 represent notable 
improvements in the validity evidence collected from the field test. In particular, the findings 
indicate stronger support for all four of the test administration claims in the Theory of Action: 


e Claim 1: TAs are prepared to administer the assessments as intended. 

e Claim 2: TAs have the resources and supports to administer the assessments. 
e Claim 3: Technology improves and facilitates the assessment experience. 

e Claim 4: Students respond to items as intended. 


The primary threats to the veracity of these claims, as identified through this Test 
Administration Study, are: 


e Continued problems of a local nature with logging students on and with students getting 
kicked-off the system. 

e The amount of time and staff required for administering PARCC assessments has been 
overly burdensome on schools’ resources. 

e Many students are having difficulty understanding the directions on the test for 
answering test questions. 


These concerns should be addressed for future operational assessments, as threats to the 
veracity of the test administration claims may undermine the validity of subsequent goals in the 
PARCC theory of action (i.e., valid scoring, reporting and use of test scores). Several 
recommendations for addressing these threats have been offered for consideration. 
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It should be noted that a memo of preliminary findings from this Test Administration Study was 
submitted to PARCC leadership in June 2015. As a result of the findings and suggestions 
provided in this preliminary memo, PARCC has already made changes to the manuals that will 
be used for the 2™ operational year, which include: 


The manuals were streamlined, reducing a great deal of redundancy. 
The Fall/Winter Test Coordinator Manual was reduced by 46 pages compared to last 
year. 

3. The Test Administrator Manuals were reduced by more than half the pages compared to 
the 2015 Spring Administration (now 59-64 pages). 

4. The structure of the manuals was changed to better align to local practices. These 
changes include restructuring the sections in the TCM and TAM to tasks before, during, 
and after testing and separating the CBT and PBT sections in the TCM. 

5. Additional content was added to further clarify procedures around using 
PearsonAccess"™' and after testing tasks. 

6. Policies were analyzed and adjusted accordingly to feedback from the field, including 
increasing the testing multiple classrooms flexibility and no longer requiring students to 
write names on scratch paper. 

7. PARCC is already working on the spring manuals, which will include great reductions to 
the student directions and the administrator scripts. 


Throughout this report, references are made to actions that PARCC has already taken or is 
taking to address concerns identified in Year 1 relevant to test administration. 
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Findings from the PARCC Quality of Test Administration Investigations 
Purpose 


The purpose of the research studies described in this report was to investigate the quality of test 
administration for the Partnership for Assessment of Readiness for College and Careers (PARCC) 
for the 2014-2015 school year (i.e., Year 1 of operational administration). Findings from these 
studies can be used to identify potential threats to the validity of PARCC test administration so 
that those threats can be addressed and reconciled for future test administrations. Throughout 
this report, threats are identified and suggestions for ameliorating those threats are provided. 
This report serves as an update to the Quality of Test Administration Study report from the field 
test (Sinclair, Deatz, Johnston-Fisher, Levinson, & Thacker, 2015), and, where appropriate, 
comparisons are made with findings from the Quality of Test Administration Study from the 
field test. 


Background 


PARCC scores are intended to be interpreted and used as an indication of whether students are 
on track in their learning to be successful in college and their careers.* This is commonly referred 
to as “college and career readiness,” or CCR. Based on the stated claims and purposes of the 
PARCC assessments as described in PARCC’s publicly available documentation, a Theory of 
Action (TOA) was developed with the end goal of college and career readiness (See PARCC 
Validity Studies Memorandum; Thacker, Sinclair, Wise, & Becker, 2014). A theory of action 
indicates the intended uses and expected impact of an assessment system and informs testable 
claims related to the interpretation of test scores (i.e., the Interpretive Argument). The evidence 
to support those claims and assumptions represents the Validity Argument. 


The TOA for PARCC (see Figure 2) is organized as a series of interim goals that lead to the end 
goal of CCR. Each interim goal is a prerequisite for supporting the subsequent interim goals. 
Again, the end goal of the PARCC assessment system is that PARCC scores in ELA and 
Mathematics provide an indication that students are, or are not, college and career ready (CCR), 
or on track to become CCR for younger students. In order to attain the end goal of CCR, each of 
the interim goals (denoted by the column headers in Figure 2) must be met. Lack of support for 
any of the interim goals potentially undermines the validity of the system to meet its end goal. 
This study, like the study from the field test, addresses the interim goal of effective test 
administration (denoted by column #4, “Administration” in Figure 2). 


1Erom PARCC website: http://www.parcconline.org/about-parcc. 
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1 2 3 4 (3) 7 
Development Implementation Jj _Administration a 


Connects to CCSS 


Determines whether 
students are CCR or on 
track to be CCR 


Measures full range of 
student performance, 
including high- and low- 
achieving students 

Tests are faster and more 
efficient to administer 


Informs instruction, 
interventions, and 
professional development 


Provides data for 
accountability, including 
measures of growth 


Incorporates innovative 
approaches, including 
technology, that are 
interactive and engaging 
Administered on range of 
devices 

Provides multiple 
measures of student 
achievement 

Includes range of item 
types 

Captures critical-thinking, 
problem-solving, and 
communication skills 
Provides timely results 
Accommodations are 
appropriate for SWD and 
ELL students 
Assessments are 
accessible to SWD 


e Subject matter, 
presentation, and 
language use is free 
of potential bias and 
is acceptable to 
students, parents, 
and other community 
members 


Items are of sufficient 
quality and rigor 
Teachers provided 
with instructional 
materials, professional 
development and 
other supports to 
enable them to 
effectively instruct 
students on CCSS 
curriculum so that 
students can 
demonstrate their 
achievement on the 
assessments 


Schools have the 
resources and 
infrastructure to 
implement and 
administer the 
assessments as 
intended 


Educators from across 
the country are 
trained by PARCC to 
become leaders and 
experts to share their 
knowledge and 
expertise within their 
community 


Teachers 
effectively 
instruct students 
on CCSS-aligned 
curriculum 


Communication 
plan clearly 
conveys to 
stakeholders the 
policies and 
practices 
essential for 
effective 
implementation 
of assessment 
system 


Students 
understand the 
format of the 
assessments and 
how to use the 
technology 


Test vendors, 
departments of 
education, 
school districts, 
and schools are 
coordinated in 
the assessment 
process 


Figure 2. The TOA for the PARCC assessment system. 


Note. Bold font indicates claims related to test score interpretation and italics indicate claims regarding impact/consequences of test scores. Reference: 
Thacker, A., Sinclair, A., Wise, L., & Becker, S. (April 10, 2014). PARCC validity studies including predictive and longitudinal studies memorandum (HumRRO 
Report 2014 No. 020). Alexandria, VA: Human Resources Research Organization. 
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Administrators 
and teachers are 
prepared to 
administer the 
assessments as 
intended 


Teachers have 
the resources 
and supports to 
administer the 
assessments 


Technology 
improves and 
facilitates the 
assessment 
experience 
Students 
respond to items 
as intended 


Timely scoring 
Rubric is 
diagnostic 

Scores accurately 
and reliably 
reflect student 
achievement on 
the assessed 
content 


Growth 
inferences 
provide accurate 
information on 
changes in 
student 
performance. 


Scores from 
multiple PARCC 
assessments 
provide both 
unique and 
complementary 
information. 


Cut score is 
indicative of 
college- and 
career-readiness 


Inferences across 
forms and years 
are 
appropriately 
comparable. 
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e Score reports 
are clear and 
easily 
understood by 
stakeholders 


e Results 
reported ina 
timely manner 


e Results 
reported in 
score reports 
are actionable 


e Students use results to 
determine if they are 
on track to graduate 
ready for college and 
careers, and to identify 
where gaps may exist 


Teachers use results to 
help inform instruction, 
and provide supports 
and interventions to 
students with readiness 
gaps 

Parents have timely 
information about the 
progress of their 
children 


States compare their 
results with other 
states to make 
decisions about their 
relative performance 
and use that 
information to better 
plan for and develop 
future workforce 


Nation compares its 
performance to other 
countries to make 
decisions about relative 
performance and use 
that information to 
better plan for and 
develop future 
workforce 
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In order to have a quality test administration, certain claims must be met. Those claims are 
depicted by the bullet points under the Administration heading in the TOA. They are: 

(a) Administrators and teachers’ are prepared to administer the assessments as intended, 

(b) Teachers have the resources and supports to administer the assessments, (c) Technology 
improves and facilitates the assessment experience, and (d) Students respond to items as 
intended. These claims served as a guide for identifying assumptions that must be met for those 
claims to be true. Those assumptions relate to the research questions guiding the investigations 
of the quality of PARCC test administration. The research questions and the investigations 
designed to address those questions were proposed by HumRRO to PARCC leadership and 
revised after multiple rounds of feedback from PARCC leadership during the PARCC field test. 
Those same research questions and investigations were used to guide the Quality of Test 
Administration Study for Year 1 of the operational PARCC test administration. 


Throughout this report, references are made to findings that relate to the administration criteria 
from the Standards for Educational and Psychological Testing (hereafter referred to as the 
Standards). The Standards provide criteria for the development and evaluation of tests and 
testing practices as well as guidelines for assessing the validity of interpretations of test scores 
for the intended test uses (AERA, APA, & NCME, 2014, p. 1). This study’s research questions and 
the Standards they address are presented next. 


?The PARCC assessment was administered by “Test Administrators” (typically teachers). Given that PARCC refers to 
the individuals administering the tests as “Test Administrators” as opposed to “teachers,” the current report treats 
all references to “teachers” in the Administration phase of the TOA as “Test Administrators.” 
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This study’s research questions and the Standards 
they address: 


1. 


10. 


To what degree do the test administrators 
(TAs) find the instructions clear, sufficiently 
detailed, and easy to follow? (addresses 
Standard 4.15) 


To what degree do the TAs follow the 
protocols and instructions? (addresses 
Standards 6.1 and 6.2) 


Do the students appear to understand the 
instructions provided to them? (addresses 
Standards 4.16 and 6.5) 


To what degree are students engaged in taking 
the test? (addresses Standards 4.16, 6.3, 6.4, 
and 6.5) 

Is there any disruptive student behavior 

during the session? (addresses Standard 6.4) 

Are there any apparent attempts to record or 

copy test materials, including the test 

questions, by students or others? (addresses 

Standards 6.6 and 6.7) 

Are there any interruptions during the 

session? (addresses Standards 6.3 and 6.4) 

a) Are there any technology-related 
problems during test administration? 

b) What type of questions, if any, do students 
ask during the test administration? (also 
addresses Standards 4.16 and 6.5) 

If any disruptions, interruptions, or other 

problems occurred, did the TAs deal with the 

issue appropriately and effectively? (addresses 

Standards 6.1 and 6.3) 

a) Were the TAs’ actions informed by 
training, by background experience, or a 
combination of the two? (addresses 
Standard 6.1) 


b) Would a different approach have been 
more effective? 

Was security of test materials maintained at 

all times? (addresses Standards 6.6 and 6.7) 

Did the test administration create minimal 

disruption to the school and staff? 
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Relevant Standards from the 
Joint Standards 


Standard 4.15: The directions for test 
administration should be presented with 
sufficient clarity so that it is possible for others 
to replicate the administration conditions under 
which the data on reliability, validity, and 
(where appropriate) norms were obtained... 


Standard 4.16: The instructions presented to 
test takers should contain sufficient detail so 
that test takers can respond to a task in the 
manner that the test developer intended. When 
appropriate, sample materials, practice or 
sample questions, criteria for scoring, and a 
representative item identified with each item 
format or major area in the test’s classification 
or domain should be provided to the test takers 
prior to the administration of the test... 


Standard 6.1: Test administrators should follow 
carefully the standardized procedures for 
administration and scoring specified by the test 
developer and any instructions from the test 
user. 


Standard 6.2: When formal procedures have 
been established for requesting and receiving 
accommodations, test takers should be 
informed of these procedures in advance of 
testing. 


Standard 6.3: Changes or disruptions to 
standardized test administration procedures or 
scoring should be documented and reported to 
test users. 


Standard 6.4: The testing environment should 
furnish reasonable comfort with minimal 
distractions to avoid construct-irrelevant 
variance. 


Standard 6.5: Test takers should be provided 
appropriate instructions, practice, and other 
support necessary to reduce construct- 
irrelevant variance. 


Standard 6.6: Reasonable efforts should be 
made to ensure the integrity of test scores by 
eliminating opportunities for test takers to 
attain scores by fraudulent or deceptive means. 


Standard 6.7: Test users have the responsibility 
of protecting the security of test materials at all 
times. 
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The next section of the report outlines the method used to collect evidence for the research 
questions on test administration. 


Method 


Three separate research investigations were designed to collect information on the interim goal 
of effective test administration. Those three research investigations are: 1) school visits to test 
administration sites to conduct observations of test administration and to conduct interviews 
with school staff, 2) surveys of Test Administrators (TAs), and 3) surveys of student test-takers. 
These three investigations provide different perspectives on the test administration, and 
thereby help to triangulate the findings related to the research questions. The research 
questions addressed by each study/data collection activity are depicted below in Table 1. 


Table 1. Research Questions Mapped to Data Collection Activity 


School Visits 


a 
Research Questions chee mel Obs? Int® 


1 To what degree do the test administrators (TAs) find the V 7 
instructions clear, sufficiently detailed, and easy to follow? 


2 To what degree do the TAs follow the protocols and 7 V 
instructions? 


3. Dothe students appear to understand the instructions V 
provided to them? 


To what degree are students engaged in taking the test? v v v 


5 Is there any disruptive student behavior during the session 
(e.g., talking during test administration instructions, teasing v 
others)? 


6 Are there any apparent attempts to record or copy test 
materials including the test questions by students or others? 


7 Are there any interruptions during the session? 


a. Are there any technology-related problems during test 

administration? 

b. What type of questions, if any, do students ask during V y os 
the test administration? 


8 _ If any disruptions, interruptions, or other problems 
occurred, did the TAs deal with the issue appropriately and v v 
effectively? 


a. Were the TAs’ actions informed by training, by V iy 
background experience, or a combination of the two? 


b. Would a different approach have been more effective? v 
9 Was security of test materials maintained at all times? 


10 Did the test administration create minimal disruption to the V V 
school and staff? 


aTA = Test Administrator. "Obs = Observation. ‘Int = Interview. 


Next, the procedures, instruments, and participants used for each of the studies are described. 
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Study 1: School Visits to Test Administration Sites 


Procedure. HumRRO researchers visited administration sites to observe the 
administration of the PARCC Performance-Based Assessment (PBA) and End-of-Year (EOY) 
operational test during the spring 2015 testing window—that is, Year 1 of operational testing. 
We based these observations on a successful protocol that HumRRO uses to ensure test 
administration quality for the National Assessment of Educational Progress (NAEP). That 
protocol is composed of detailed observation checklists that trained HumRRO staff use to 
document task completion during a sample of test administrations. HumRRO revised the 
protocol based on the PARCC Test Administrator Manual, guidance from PARCC leadership, and 
feedback received after using the protocol during the PARCC field test study in 2014. 


We also conducted short debriefing interviews with school Test Administrators (TAs) or Test 
Coordinators (TCs) directly after observing a test session if time permitted. 


After observation and interview protocols were reviewed and approved by PARCC leadership, 
but prior to visiting schools, HumRRO conducted internal training for selected staff. Although 

the HumRRO research staff selected for the school visits were experienced, it was essential to 
provide them with information tailored to this project. HumRRO conducted the training with 

staff via teleconference, with interested Pearson and PARCC staff in attendance. The training 

covered the following topics: 


e Anoverview of PARCC and the 2015 operational test administration 

e PARCC assessment system and materials 

e What to expect when observing 

e Observation checklist (detailed review) 

e Interview debriefing protocol (detailed review) 

e Observer conduct during school visits 

e Coordination with the sampled schools 

e What to take with you (e.g., observation forms, job aid, PARCC manual excerpts) 


A key aspect of the training was a detailed review of the observation checklist (described 
below) to ensure that observers had a common understanding of each “task” (i.e., the activities 
performed by the TAs during testing) for rating consistency across observers. Observers learned 
what they would see when the task is conducted, or in some cases, how they would know if a 
task is completed. This is an important discussion since it can be unclear at times which rating is 
most appropriate to use. For example, for the task, “Monitor student progress using 
administration computer,” if the observer can see the TA looking at TestNav usage screens 
during the assessment, then the observer was trained to rate the task as “Met” on the 
observation checklist. 
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Additionally, procedures for interacting with the schools, school personnel, and students were 
reviewed during the training. Observers were trained not to move around the testing room, 
interfere with or provide guidance to test administrators, or interact with students. The 
observers were instructed to be as unobtrusive as possible during the site visits. 


The HumRRO researchers assigned to visit schools were directed to contact the school’s PARCC 
test coordinator to verify the testing date and time, address any last minute logistical details, 
and to provide assurance that we were not visiting to evaluate the school or their staff. 
Observers arrived at the schools wearing their PARCC photo ID cards. They followed the school 
sign-in procedures, and they were accompanied by the test coordinator or TA to the testing 
room. The observers positioned themselves near the administration computer when possible 
and remained seated (once students arrived) throughout the test. They used applicable 
resources (excerpts from the TA manual or a job aid?) and entered their observation ratings and 
notes into the observation form using either their laptops or paper copies of the observation (if 
a school requested that no laptops be used in the testing room). Afterwards, they conducted a 
debriefing interview with the TA and/or Test Coordinator (TC), if possible. 


Instruments. HumRRO developed two data collection instruments for use during test 
administration observations. The first instrument contained a list of activities (“tasks”) 
performed by the TA that were based on the guidance for test administration in the PARCC Test 
Administrator (TA) Manual. There was one version for observing paper-based tests (PBTs) and 
another version for observing computer/tablet-based tests (CBTs). Observers rated each task 
on a rating scale (described below). This instrument included space for observer notes so that 
observers could provide the rationale for their rating, as well as other observations not 
specifically linked to a particular task. The second instrument, a semi-structured interview 
protocol, was developed for post-assessment interviews. All instruments were reviewed and 
approved by PARCC leadership prior to their use. 


Test administration observation checklist. We designed the observation checklist as an internal 
data collection form targeting observable test administration activities. The purpose of the 
form was to record whether specific administration tasks were conducted and to document 
exceptions to established procedures, if any were found. HumRRO used the PARCC Computer 
Based Testing (CBT) Test Coordinator, Administrator, Accessibility Features, and 
Accommodations manuals to identify the observable tasks included in the checklist. 


The observation checklists were Excel® documents that HumRRO observers used to enter data, 
which were uploaded to Statistical Analysis Software (SAS™) to facilitate aggregated reporting 
across observations. The observation checklists (for CBT and PBT) can be found in Appendix A. 


3This was a resource developed by HumRRO and PARCC staff containing the procedures used to administer 
accommodations. 
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The observers were trained to rate each task on a five-point rating scale. The rating scale anda 
description of each scale anchor are presented below in Table 2. 


Table 2. Rating Scale for Observation Checklist 


Rating Description 


Met The task was completed. 


Partially Met Some parts of the task were not completed. 


1 

2 

3 Did Not Meet The observer knows that the task was not completed. 

4 Not Observed The task was likely completed but was not directly seen by the observer. 
5 


Not Applicable The task is not intended to be completed in certain situations. 


To support the use of the observation checklist when observing accommodation sessions, 
HumRRO developed an accessibility and accommodation guide. Since the tasks the TAs follow 
vary for each testing accommodation that students require, as specified in their Individualized 
Education Program (IEP), listing all possible combinations in the checklist was not feasible. 
Observers used the guide, based on PARCC accommodation documents (e.g., manual, training 
slides), to inform their ratings while observing an accommodation session. 


Test coordinator and administrator debriefing interview protocol. To obtain additional 


information regarding the overall test administration experience of the TA, TC, and other school 
or district staff (e.g., other teachers helping the TA, technology personnel), we developed an 
interview protocol form as a guide for conducting post-assessment interviews. If time 
permitted, HumRRO observers asked the TA and other applicable staff about their experiences 
and feedback regarding the testing process, training materials, and support from PARCC 
leadership. See Appendix B for the debriefing interview protocol, one for CBT and PBT 
administrations. 


Participants and School Visits. HumRRO sampled schools in the following 10 PARCC 
states: Arkansas, Colorado, District of Columbia, Illinois, Maryland, Massachusetts, New Jersey, 
New Mexico, Ohio, and Rhode Island. PARCC leadership provided HumRRO contact information 
for a point of contact (POC) at each state department of education. The goal was to complete 
observations in 100 schools, 10 from each state, distributed across the PBA and EOY test 
administrations. Additional factors that were considered when selecting and scheduling schools 
included: CBT and PBT administrations, academic content areas (Mathematics and 
ELA/Literacy), grade spans (elementary, middle, and high schools), and geographic locale. 


School recruiting was a multistep process. First, for PBA administrations, HumRRO determined 
the geographic area we would visit for each state and asked our state level POC to provide us 
contact information for the identified district POCs and to send a pre-emptive email to the 
district POCs to inform them that they would be contacted by HumRRO. We selected 5-10 
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schools in the identified districts and provided that list to the district POC to help us recruit 
schools by sending an email to the schools’ TCs and copying HumRRO on that email. We 
followed up that email to the schools with a document introducing HumRRO and this PARCC 
study. This document also included a request to observe a PARCC test administration session 
and to obtain the school’s testing schedule. We also reinforced our commitment to observe 
testing as unobtrusively as possible and we assured schools that our findings would be reported 
in aggregate only. We informed schools that we were not evaluating the school or school staff, 
but focusing only on the administration of the PARCC operational test for quality assurance and 
process improvement purposes. 


During the 2014-2015 school year, we visited a total of 99 schools and observed 109 test 
sessions (see Table 3). As explained in the PARCC-approved study plan, we conducted a brief 
debriefing interview with the TAs and/or TCs directly after the observed test administration 
when time permitted. We anticipated that for operational testing most sessions would run 
back-to-back and that TAs would not have time for an interview. However, of the 99 completed 
school visits, we were able to conduct 88 interviews (43 PBA and 45 EOY) at 85 different 
schools. There were three schools where we observed more than one test administration, for 
example a CBT and PBT administration, with different TAs. 


Table 3. Summary of PARCC School Visits 


PBA Breakout EOY Breakout 


Number of Schools parabens ct MECHEL at 
Observations Observations 
State PBA EOY Total CBT PBT Total Totallnt® CBT PBT Total Total Int? 
Arkansas 5 5 10 3 2 5 4 5 1 6 2 
Colorado 5 5 10 6 0 6 5 5 0 5 3 
District of Columbia 5 5 10 4 1 5 4 5 0 5 3 
Illinois 4 6 10 3 2 5 5 4 2 6 5 
Maryland 6 6 12 9 0 9 4 7 1 8 6 
Massachusetts 4 3 7 a 3 4 4 2 1 3 4 
New Jersey 5 5 10 5 0 5 6 5 0 5 5 
New Mexico 5 6 11 5 0 5 5 4 1 5 7 
Ohio 0 9 9 0 0 0 0 12 0 12 10 
Rhode Island 5 5 10 4 1 5 6 3 2 5 0 
Total 44 55 99 40 9 49 43 52 8 60 45 


Note. There was not a one-to-one correspondence between the number of schools and the number of 
observations or interviews conducted at a school. In several instances, multiple observations/interviews were 
conducted at a school. Consequently, the sum of the number of observations/interviews does not always equal the 
number of PBA schools and the number of EOY schools. 

Int = Interview. 


Additional information regarding the distribution of session types, grade spans, and subjects for 
both PBA and EOY observations are presented in Table 4. 
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Table 4. Summary of School Visit Demographic Make-up 


Session Type Observed Grade Span Subject 
State Reg? Accom? Makeup ES MS HS ELA Math Alg/Geo 
Arkansas 7 4 0 4 4 3 6 5 0 
Colorado 11 0 0 4 4 3 6 4 1 
District of Columbia 6 1 3 5 3 2 5 5 0 
Illinois 9 1 1 9 2 0 6 5 0 
Maryland 12 5 0) 5 5 7 10 3 4 
Massachusetts 6 1 0 6 1 0 4 3 0 
New Jersey 7 3 0 3 5 2 5 4 1 
New Mexico 7 2 1 7 2 1 3 6 dl 
Ohio 9 2 1 3 8 1 6 4 2 
Rhode Island 7 3 0 5 3 2 2 8 0 
Total 81 22 6 51 37 21 53 47 9 


"Reg = Regular test session. "Accom = Accommodation (either in regular or separate sessions). 


Study 2: Test Administrator Surveys 


Procedure. An online Test Administrator Survey was completed by TAs following 
administration of the PARCC PBA and EOY assessments. The primary purpose of the survey was 
to collect information from the perspective of TAs on the effectiveness of the training they 
received and to identify potential problems that arose during test administration. The TAs were 
provided a web link to the online survey in their TA manual. They were instructed to log on and 
complete the survey after test administration was completed. The TAs responded to a separate 
survey for computer/tablet-based test (CBT) administration and for paper-based test (PBT) 
administration. The online test administrator survey was available from February 16, 2015 to 
June 12, 2015, which corresponded to the first day of PARCC test administration and one week 
after the last day of PARCC test administration. 


Instruments. Two separate TA Surveys were developed—one for TAs administering CBTs 
(see Appendix C) and an edited version for TAs administering PBTs (see Appendix D). Survey 
items related to technology were not relevant to TAs administering PBTs; hence, the need for 
different versions of the surveys. Both surveys were based on the surveys developed for the 
PARCC field test. Both the CBT and PBT TA Surveys were on-line surveys that were hosted on 
the HumRRO platform. The web links for the surveys were included in the Test Administration 
manuals along with instructions for TAs to complete the survey within one week after 
completing test administration. The TA manuals stated that, “If a Test Administrator is 
administering both the PBA and the EOY, he or she should complete the survey once after each 
administration.” The first question on the survey asked TAs to identify which type of 
assessment they had most recently administered—PBA or EOY; this question did not appear on 
the field test version of the TA Survey. For the operational TA Study, the PARCC leadership 
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requested separate breakout analyses for the PBA administration and the EOY administration. 
Consequently, this initial question was added to both the CBT TA Survey and the PBT TA Survey 
for Year 1 of the operational test administration. 


The items on the surveys were all close-ended survey items with the exception of two open- 
ended items—one of which asked for suggestions on improving the PARCC online trainings and 
a second which asked for recommendations for improving test administration policies and 
procedures. Per the approved Study Plan for the Quality of Test Administration Study, the 
open-ended responses to these survey items have been provided to PARCC leadership and to 
the states for analysis. 


Participants. There were 28,031 cases in the raw data file for the CBT TA Survey. 
Respondents who did not complete any items on the survey or who only responded to the 
background survey items and/or who only responded to one or two other items on the survey 
were removed from the data set. This resulted in 21,777 respondents for the CBT TA Survey, 
with 10,545 of those responding for the PBA administration of the CBT and 11,232 of those 
responding for the EOY administration of the CBT. 


There were 5,043 cases in the raw data file for the PBT TA Survey. Likewise, respondents who 
did not complete any items on the survey or who only responded to the background survey 
items and/or who only responded to one or two other items on the survey were removed from 
the data set. This resulted in 3,805 valid respondents to the PBT TA Survey, with 2,336 of those 
responding for the PBA administration of the PBT and 1,469 of those responding for the EOY 
administration of the PBT. 


The response breakout by state, mode of administration, and assessment type is presented in 
Table 5. As seen in Table 5, Arkansas, Illinois, and Massachusetts were the states with the 
largest percentages of TAs responding to the TA Survey for CBT administration, and Illinois and 
Massachusetts were the states with the largest percentages of TAs responding to the TA Survey 
for PBT administration. 


Submitted September 28, 2015 Page 11 


PARCC Quality of Test Administration 


Table 5. TA Survey Response Breakout by State and by Mode of Assessment Administration 


CBT PBA CBT EOY PBT PBA d= i 0) f 

State N % N % N % N % 
Arkansas 2,358 22.4 2,838 25.4 89 5.1 189 14.5 
Colorado 136 1.3 183 1.6 18 1.0 11 0.8 
District of Columbia 55 0.5 30 0.3 2 0.1 2 0.2 
Illinois 2,264 21.5 1,803 16.1 440 25.3 160 12.3 
Massachusetts 693 6.6 752 6.7 662 38.0 449 34.5 
Maryland 555 5.3 1,056 9.4 26 1.5 154 11.8 
Mississippi 92 0.9 72 0.6 40 2.3 -- -- 
New Jersey 1,920 18.3 3647 32.6 31 1.8 173 13.3 
New Mexico 1256 12.0 330 2.9 57 3.3 44 3.4 
Ohio 463 4.4 433 3.9 239 13.7 94 7.2 
Rhode Island 718 6.8 44 0.4 136 7.8 25 1.9 
All States 10,5458 100.0 11,232 100.0 2,336° 100.0 1,469 100.0 


Note. Percentages are based on the total number of valid (non-missing) responses for each state. Number of 
respondents for which a response to the location question was missing was n = 35 for CBT PBA; n = 44 for CBT EOY; 
n = 5964 for PBT PBA; and n = 168 for PBT EOY. 

lncludes the 35 cases that did not indicate their state. "Includes the 44 cases that did not indicate their state. 
‘Includes the 596 cases that did not indicate their state. ‘Includes the 168 cases that did not indicate their state. 


Study 3: Student Surveys 


Procedure. Upon completion of the CBT EOY PARCC assessments, students had the 
opportunity to complete a survey about their test-taking experience’. Different surveys were 
administered following completion of Mathematics and ELA assessments. There was 
considerable overlap in the content of the items on the Mathematics and ELA Student Surveys, 
although each contained a subset of items specific to the content area being assessed (e.g., 
question about entering symbols for the Mathematics survey and question about moving 
between passages for the ELA survey). 


Instruments. HumRRO researchers developed draft Student Surveys (one for 
Mathematics and one for ELA) based on earlier versions of the Student Surveys that were used 
for the PARCC field test (Sinclair, et al., 2015). The most substantive edit from the field test to 
operational testing was to replace the “select all that apply” formatted items to “select one” 
formatted items in order to facilitate data analysis and interpretation. The survey content was 
reviewed, edited, and approved by PARCC leadership. 


“There were a disproportionately large number of respondents who did not indicate their state for the PBT TA 
Survey following PBA administration. Additional scrutiny was conducted to determine if the high number of 
missing responses to the location question for the PBT PBA might have been due to an issue with the survey; 
however, no evidence of any systematic issue with the survey was found that might explain why so many 
respondents skipped the location question on this survey. 

*The Student Surveys were connected to the TestNav system, and administered and collected by Pearson. The 
system was configured such that the Student Surveys were only available at the end of the CBT EOY assessments. 
As such, Student Survey data was not collected following the PBA CBTs, nor from any of the students taking the 
PBTs. 
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The items that appeared on the Student Survey for the CBT EOY for Mathematics are provided 
in Appendix E. The items that appeared on the Student Survey for the CBT EOY for ELA are 
provided in Appendix F. The items on the surveys were all close-ended survey items with the 
exception of the final question on the survey, which asked to students to comment about 
anything that was confusing or unclear on about the test. Per the approved Study Plan for the 
Quality of Test Administration Study, the open-ended responses to this survey item have been 
provided to PARCC leadership and to the states for analysis. 


Participants. Participants were the students who took the CBT EOY assessments in 
Mathematics and ELA. Table 6 below identifies the number and percentage of students 
responding to the surveys in each state and for all states combined. As seen in Table 6, the 
states with the largest percentages of Student Survey responses were Illinois and New Jersey. 


Table 6. Student Survey Response Rates 


Mathematics EOY CBT 3 0) Ae =3 0 


State n % n % 
Arkansas 256,924 7.21 266,298 7.60 
Colorado 354,923 9.96 381,752 10.90 
District of Columbia 30,933 0.87 30,799 0.88 
Illinois 703,517 19.74 713,566 20.37 
Maryland 374,667 10.51 335,775 9.58 
Massachusetts 127,621 3.58 127,410 3.64 
Mississippi 210,443 5.91 202,001 5.77 
New Jersey 732,588 20.56 755,599 21.57 
New Mexico 187,726 5.27 188,638 5.38 
Ohio 525,142 14.74 443,902 12.67 
Rhode Island 59,139 1.66 57,734 1.65 
All States 3,563,623 100.0 3,503,474 100.0 
Results 


The findings from the three studies are presented in this section according to the research 
question they address. It is should be noted that there are several instances of overlap among 
the research questions. For example, Research Question 7 (Are there any interruptions during 
the session?) overlaps with Research Question 4 (To what degree are students engaged in 
taking the test? [i.e., interruptions often negatively impact engagement]). Consequently, one 
set of findings may be relevant to multiple research questions. Thus, there is some redundancy 
in the reporting of results across research questions; however, in order to reduce the length of 
the report some results are presented for one research question, but are identified as being 
relevant to other research questions. 
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The full set of item-level results for the TA Surveys (with breakouts by state) can be found in 
Appendix G (CBT version) and Appendix H (PBT version). The full set of item-level results for the 
Student Surveys (with breakouts by state, and by PBA and EOY) can be found in Appendix | (CBT 
EOY Math version) and Appendix J (CBT EOY ELA version). 


For the TA Survey, it should be noted that the TAs were not assigned a unique login ID for 
responding to the survey. Consequently, while most of the TAs who responded to the survey 
after administering the PBA are the same TAs who responded to the survey after administering 
the EOY, we do not have a true matched/paired sample. Consequently, we did not conduct a 
matched-pair t-test to investigate whether there are statistically significant differences 

between TAs’ PBA responses and their EOY responses. Moreover, since the two samples are not 
independent samples it would not be appropriate to conduct an independent samples t-test 
comparing PBA responses and EOY responses. Therefore, no statistical significance tests were 
conducted to compare TA survey responses for PBA vs. EOY administrations. 


For the Student Survey, it should be noted that the survey was configured such that it was 
presented to students immediately following completion of the mathematics and ELA 
assessments, and therefore the student survey responses are considered to represent the full 
population. Consequently, statistical significance tests were not conducted comparing students’ 
responses on the mathematics and ELA surveys (i.e., statistical significance tests are used to 
indicate how likely it is that the results are representative of the full population). 


For both the TA Survey and Student Survey, the majority of the survey items were rated on 
nominal or ordinal scales (e.g., “Sufficient time to finish the test?” with response options of: 
Finished very early, Finished on time, Rushed to finish, Did not finish). Consequently, for both 
the TA Surveys and the Student Surveys results are presented in terms of frequency counts and 
percentages, and comparisons between surveys are presented graphically with a focus on 
substantive (i.e. practical) differences as opposed to statistically significant differences. 


Findings for Research Question 1: To what degree do the TAs find the instructions clear, 
sufficiently detailed, and easy to follow?® 


The claim in the theory of action most directly related to this research question is, “Teachers 
have the resources and supports to administer the assessments” (claim 2). The instructions in 
the Test Administrator Manual serve as the TAs’ primary resource for administering the 
assessments. An important assumption underlying this claim is that this resource (i.e., the TA 
Manual) is sufficiently clear and detailed, and easy to implement. To verify the veracity of this 
assumption the TA Surveys asked TAs to rate their agreement with the statements, “The 


This research question focuses on the Test Administrator Manual. For information on the effectiveness of the 
Accessibility Features and Accommodations Manual in guiding TAs in administering AFAs see the results for 
Research Question 2. 
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policies and procedures within the Test Administrator Manual were easy for me to understand,” 
and “The instructions (including the scripts) within the Test Administrator Manual were easy for 
me to implement.” The findings presented in Figure 3 indicate that the vast majority of TAs for 
Year 1 (over 80%) agreed that the policies and procedures in the TA Manual were easy for them 
to understand. This is an increase from the TA survey results from the field test where 71% of 
CBT administrators and 67% of PBT administrators agreed that the policies and procedures in 
the TA Manual were easy for them to understand. Similarly, the vast majority of TAs for Year 1 
(over 80%) agreed that the instructions (including the scripts) in the TA Manual were easy for 
them to implement, whereas results from the TA Survey from the field test indicated that less 
than two-thirds of TAs (55% - 63% for PBT and CBT, respectively) agreed that the instructions in 
the TA Manual were easy for them to implement (see Figure 4). 


TA Survey: "The policies and procedures within the Test 
Administrator Manual were easy for me to understand." 


100.0% 
90.0% 
m CBT PBA 
80.0% N = 10,545 
m CBT EOY 
0, 
70.0% N = 11,232 
wy 00.0% m PBT PBA 
2 N = 2,336 
&) 50.0% 
= m PBT EOY 
® 10.0% N= 1,469 
30.0% Field Test CBT 
: n= 4,635 
20.0% m Field Test PBT 
n = 1,687 
10.0% 
0.0% 


Figure 3. Percentage of TAs agreeing that the TA Manual is easy to understand. 


Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect 
the combined percentage of “Agree” and “Strongly Agree” responses. 
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TA Survey: "The instructions (including scripts for 
administering the assessment) within the Test Administrator 
Manual were easy for me to implement." 


100.0% 
90.0% g4.9% 871% — ay 6% = 86.0% 
mCBT PBA 
80.0% N = 10,545 
70.0% m CBT EOY 
63.3% N = 11,232 
y 00.0% 55.2% m PBT PBA 
hed N = 2,336 
& 50.0% 
: m PBT EOY 
* 40.0% N= 1,469 
m Field Test CBT 
ae n= 4,635 
20.0% m@ Field Test PBT 
n= 1,687 
10.0% 
0.0% 


Figure 4. Percentage of TAs agreeing that the instructions in the TA Manual were easy to implement. 
Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect 


the combined percentage of “Agree” and “Strongly Agree” responses. 

Another data source for informing Research Question 1 were the debriefing interviews with 
TAs, which were conducted after observing TAs administering the PARCC test to students. The 
TAs were asked two questions that directly relate to this research question: 


1. Was the test administration manual beneficial in preparing for the test? 
2. Were the scripts easy to follow and provide the right amount of information for 
students? 


The responses were coded into themes and there were no particular differences in responses 
from TAs administering PBA and EOY assessments. The themes for the first question are 
presented in Table 7. The first theme pertained to the amount of information provided in the 
TA manual. TAs in 33 of 88 interviews (38%) described the manual as “helpful,” and “...provided 
the information needed.” However, TAs in 31 interviews (35%) indicated the TA manual 
contained too much extra information that did not directly guide the test administration task; 
for example, some indicated the manual was “far too dense to retain necessary information.” As 
a result, TAs in 15 interviews (17%) reported that their districts created summary support 
documents, some with screen capture images, that helped TAs ensure that they covered the 
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key tasks during test administration. The organization of information in the manual was 
described positively in 22 interviews (25%) in that the information was logically presented, 
directions were easy to follow, and it was easy to access. In nine interviews, TAs felt the 
information was not presented sequentially or intuitive to find information they needed. Other 
comments received during the interviews indicated the manuals were not received in a timely 
manner (i.e., the day before testing began) and that managing several updates to the manual 
after testing had begun was difficult. Additional suggestions obtained from TAs during the 
debriefing interview for improving the TA Manual are presented in Appendix K. 


Table 7. Usefulness of TA Manual: Themes Identified from Interviews with TAs 


Theme Sources? References® Sample Responses 


Liked the "fill in the blank" statements for the number 
of minutes and grade level, etc. 


inben nation 33 36 e TAs found it self-explanatory and they did not have too 
adequate ; 
many questions 
e Yes, the manual was especially helpful in shepherding 
brand new teachers. 
Information not e It is fairly lengthy and could be condensed. 
31 37 : eee 
adequate e A lot of unnecessary information; kind of wordy. 
e Manuals could be "scaled down." 
oo e Seemed well-organized and thorough. 
Organization . : 
d 22 26 e Really easy to access; well organized. 
aueMars e Helpful to have a glossary to describe distinctions 
between terminology 
0 oe k 9 10 e Not very intuitive. 
tganizarlon' wed e TC and TA manuals were not organized in similar ways. 
e Online manual needs Table of Contents links 
e Would like to see simple steps for teacher, especially 
Suul with troubleshooting. 
VBP eimietite e The most helpful document was a step-by-step guide 
materials created by 15 16 


for conducting the assessment that was prepared by 
the district. 

The TC provided them with screen shots of steps, which 
the TAs said were helpful. 


indicates number of interviews providing information about that topic. "Indicates number of notes and/or 
comments made about that topic. 


districts 


Table 8 summarizes information received during the debriefing interviews with TAs regarding 
the usefulness of the test administration scripts included in the TA Manuals. In nearly three- 
fourths (65 of 88) of the interviews, the TAs and TCs described the scripts as effective, with 
comments such as “comparable to scripts on other assessments I’ve administered” and “the 
scripts provided the necessary information to students.” Staff in only 15 interviews (17%) stated 
that the scripts were “redundant” and “streamlining needed,” particularly for students 
completing multiple sessions and tests. Additional clarifications on navigating between sessions 
and ending sessions and exiting the test were requested by TAs in seven interviews (8%); this 
was a prevalent theme in the field test study. Consequently, given that this topic was only 
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mentioned in 8% of the interviews in Year 1, this suggests that improvements have been made, 
although additional fine-tuning may still be warranted. Comments obtained in seven interviews 
(8%) indicated that TAs/TCs were unsure why they needed to test the sound through the 
headphones for test sessions that did not include audio. 


Also, there were comments from four interviews regarding accommodations. TAs suggested 
that there should be a separate accommodation script with additional cues for navigation and 
clearer guidance for TAs on procedures such as read aloud. Also, there were two TAs who 
commented that there were no instructions in the script for proceeding to the Student Survey 
and also that the text-to-speech accommodation does not work with the Student Survey. 
Additional suggestions obtained from TAs during the debriefing interview for improving the 
script are presented in Appendix K. 


Table 8. Usefulness of Scripts: Themes Identified from Interviews with TAs 


Theme Sources? References® Sample Responses 


It is lengthy but overall it is important to restate and 
reinforce the directions. 

Sequential, made sense and to the point. 

The scripts had the right amount of information. 
The boxes and bold print were easy to follow. 
Maybe a little lengthy. It is hard to listen to all the 
commands at the same time. 

Repetitive 

The script intro for 3rd graders is too long for this age; 
the kids get antsy before the test even begins. 


Script effective 65 68 


Script needs 


: 15 17 
improvement 


Also, as written, the script does not do enough to keep 


students from submitting their test too early (before all 


ae sessions completed). 
Navigating between P ) 


sessions and ending 
sessions and exiting 
the test 


7 10 mistakenly administered two sessions in one day. 
There is no clear way to end a group that is already all 
finished as happened in this session. May be easier to 
have the people log out at the same time so you only 
have to read the instructions once. 


don't have sound? This is confusing! 
Feels a little odd to have to test the headphones if 
students aren't using them. 


Testing sound and 
headphones 


N 
ee) 
e 


indicates number of interviews providing information about that topic. "Indicates number of notes and/or 
comments made about that topic. 

When compared to the findings from the School Visits Study from the field test, the changes 
made in the organization and content of the scripts since the field test appear to have been 
beneficial. For the field test, in nearly half of the interviews, the TAs provided unfavorable 
comments about the script; for example, the TAs commonly indicated that the script was 
difficult to follow and repetitive, and that the terminology was unclear (e.g., sessions vs. 
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section); whereas positive comments about the scripts were provided in nearly three-fourths 
(65 of 88) of the interviews for the Year 1 study. 


Collectively, these findings suggest that the actions implemented by PARCC since the field test 
to improve the TA Manual such as, shortening the TA Manual by reducing redundancies, adding 
grade band scripts, adding checklists of tasks, adding additional graphics and icons to increase 
user understanding, clarifying terminology, etc. (see PARCC Field Test: Lessons Learned, PARCC, 
2015) were beneficial and favorably received by the TAs. To the extent that the actions taken by 
PARCC since the field test have improved the clarity of instructions and the ease of 
implementing instructions, which the findings presented here suggest that they have, then 
adherence to Standard 4.15 has been further strengthened since the field test. Possible areas 
for further refinement include providing additional clarification on navigating between sessions 
and exiting the test (CBT), further streamlining the scripts to reduce redundancies for students 
completing multiple sessions and tests, and providing the finalized TA Manuals earlier (it should 
be noted that as a result of this advice PARCC provided manuals for the 2015 Fall Block 
administration 60 days in advance). 


Findings for Research Question 2: To what degree do the TAs follow the protocols and 
instructions? 


The claim in the theory of action most directly related to this research question is, “TAs are 
prepared to administer the assessments as intended” (claim 1). For this claim to be true at least 
two assumptions must be met: (a) administrators must be trained on how to administer the 
assessments, and (b) that training must be effective. To investigate the veracity of these 
assumptions, the TA Survey (CBT and PBT versions) asked the TAs a series of questions about 
the effectiveness of the training and training materials they received prior to administering the 
assessments. The trainings and training materials for administering CBTs and PBTs differed 
somewhat. Consequently, there were some differences in survey questions on the CBT TA 
Survey and the PBT TA Survey regarding training, and, thus, findings are presented separately. 


Findings on the usefulness of the training modules for TAs administering CBTs are presented in 
Figure 5’. Overall, results were quite similar across all training modules. For those who 
indicated that the training was applicable to them, approximately two-thirds agreed that the 
modules effectively prepared them to administer the CBT PARCC assessments. It should be 
noted that a sizeable percentage of respondents to the CBT TA Survey (35%-40%) indicated that 
the following trainings were “not applicable” to them: Personal Needs Profile (PNP) Training, 
Proctor Caching & TestNav Configuration Training, Student Registration Import Training, and 
Technology “Readiness” for Schools and Districts Training. 


7Due to differences in training modules and due to differences in the format and wording of the items on the 
surveys, comparisons with field test survey results are not presented. 
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CBT TA Survey: "This training module effectively prepared me to administer the PARCC 
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Figure 5. Percentage of TAs agreeing that training modules prepared them to administer CBT PARCC assessments. 

Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect the combined percentage of “Agree” and 
“Strongly Agree” responses. Respondents indicating “Not Applicable” are excluded from these results. Across this set of items, n-counts ranged from n = 6,430 
to 9,098 for CBT PBA and from n = 6,943 to 9,664 for CBT EOY. 
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The TAs administering CBTs were also explicitly asked to indicate the extent to which they 
agreed that the PARCC online trainings prepared them to resolve basic problems related to 
technology (e.g., students logging in, exiting the test, etc.). This same question was also asked 
on the field test TA Survey. Figure 6 below shows that just over half of the TAs (56%) 
administering the CBTs during Year 1 agreed that the PARCC online trainings prepared them to 
resolve basic problems related to technology; this is a modest increase from the field test in 
which 49% of TAs indicated that the PARCC online trainings prepared them to resolve basic 
problems related to technology. 


CBT TA Survey: "The PARCC online training(s) prepared me to 
resolve basic problems related to technology." 
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Figure 6. Percentage of TAs agreeing that PARCC online trainings effectively prepared them to resolve 
basic problems related to technology. 


Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect 
the combined percentage of “Agree” and “Strongly Agree” responses. Respondents indicating “Not Applicable” are 
excluded from these results. 


Findings on the usefulness of the training modules for TAs administering PBTs are presented in 
Figure 7. For those that indicated the training was applicable to them, three-fourths or more 
(75% - 83%) agreed that the Introduction to PARCC Training Modules, the Accessibility Features 
& Accommodations Training Module, and the Administration of Paper-based Assessments 
Module effectively prepared them to administer the PBT PARCC assessments. 


Approximately two-thirds agreed that the trainings on PearsonAccess"™*, Student Registration 
Import, and Student Readiness Resources helped prepare them to administer the PBT PARCC 
assessments (which were similar to findings from the CBT TA Survey). It should be noted that a 
sizeable percentage of respondents to the PBT TA Survey (41% - 53%) indicated that these 
trainings--i.e., PearsonAccess"™t Training, Student Registration Import Training, and Student 
Readiness Resources for PARCC Training--were “not applicable” to them. Of these only 
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PearsonAccess"™™' Training and Student Registration Import Training are not applicable for TAs 
administering PBTs. The student readiness resources for PARCC Training is appropriate for TAs 
administering PBTs, which suggests that there might be a disconnect in teachers’ awareness of 


which trainings are relevant. 
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Figure 7. Percentage of TAs agreeing that training modules prepared them to administer PBT PARCC assessments. 

Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect the combined percentage of “Agree” and 
“Strongly Agree” responses. Respondents indicating “Not Applicable” are excluded from these results; across this set of items, n-counts ranged from n = 1,151 
to 1,885 for PBT PBA and from n = 755 to 1,228 for PBT EOY. 
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All TAs, regardless of whether they administered CBTs or PBTs, were also asked to rate the 
extent to which they agreed that the training they received from their state department of 
education (DOE), from their district, and/or from their school effectively prepared them to 
administer PARCC assessments. Those results are presented in Figure 8. Of those who indicated 
the training was applicable to them, the vast majority (73% - 88%) agreed that the district 
training and school training effectively prepared them for administering the assessments with a 
slightly higher percentage agreeing that the school training was effective (82% - 88%). Of the 
TAs who indicated that the state DOE training was applicable, approximately half (43% - 55%) 
agreed that it effectively prepared them to administer the PARCC assessments. It should be 
noted that a sizable percentage of TAs (35% and 40% for CBT and PBT, respectively) indicated 
that the state DOE training was “not applicable” to them. 


TA Survey: "This training effectively prepared me to administer 
the PARCC assessments." 
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Figure 8. Percentage of TAs agreeing that state, district or school training effectively prepared them. 
Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect 


the combined percentage of “Agree” and “Strongly Agree” responses. Respondents indicating “Not Applicable” are 
excluded from these results; across this set of items, n-counts ranged from n = 6,944 to 10,151 for CBT PBA, from 
n= 7,391 to 10,813 for CBT EOY, from n = 1,404 to 2,219 for PBT PBA, and from n = 901 to 1,410 for PBT EOY. 

The ways in which TAs were able to access and interact with PARCC content prior to 
administration may also affect their preparedness to administer assessments as intended. One 
of the findings that stemmed from the Quality of Test Administration Study from the field test 
was that TAs wanted an opportunity to review PARCC content (e.g. sample items, practice test) 
prior to administering the assessments. They felt that this would make them more comfortable 
and knowledgeable about the test administration protocol. Consequently, for the operational 
assessment TAs were provided additional opportunities to review PARCC content prior to 


administration. The results presented in Figure 9 indicate that roughly two-thirds of TAs for 
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both test formats (CBT and PBT) and for both test types (PBA and EOY) reviewed sample items 
prior to administering the PARCC assessments. Similarly, two-thirds of TAs administering the 
CBT reported that they reviewed PARCC tutorials prior to administration; although only about 
one-third of TAs administering the PBT reported that they reviewed PARCC tutorials prior to 
administration. A similar pattern emerged for completing a practice test, although the 
difference was not quite as pronounced. Very few TAs (8% - 17%) reported that they “did not 
practice” prior to test administration. 


TA Survey: "How did you, as a Test Administrator, work with 
PARCC content prior to administration?" (Select all that apply) 


100.0% 
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Figure 9. Percentage of TAs indicating how they worked with PARCC content prior to administration. 
Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 


differentiate “missing” (i.e., skipped) vs. “not selected” responses. 

Finally, TAs administering the CBT were asked to indicate the extent to which they agreed with 
the statement, “Completion of the PARCC tutorial helped me to better understand the tools 
and functionalities of the TestNav system.” Of those for whom this question was applicable, 
more than 80% agreed or strongly agreed with this statement (for both PBA and EOY)—see 
Figure 10. Similarly, TAs administering the PBT were asked to rate the extent to which they 
agreed with the statement, “Completion of the PARCC paper-based student tutorials helped 
better prepare me for administering the assessment.” Of those for whom this question was 
applicable, nearly three-fourths agreed or strongly agreed with this statement (for both PBA 
and EOY)—see Figure 11. 
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CBT TA Survey: "Completion of the PARCC tutorial 
helped me to better understand the tools and 
functionalities of the TestNav system." 
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Figure 10. Percentage of TAs agreeing that the PARCC tutorial helped them understand the TestNav 


system. 

Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect the 
combined percentage of “Agree” and “Strongly Agree” responses. Respondents indicating “Not Applicable” are excluded 
from these results. 


PBT TA Survey: "Completion of the PARCC paper- 
based student tutorials helped better prepare me for 
administering the assessment." 
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Figure 11. Percentage of TAs agreeing that completion of PARCC paper-based student tutorials helped 


better prepare them to administer the assessment. 

Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect the 
combined percentage of “Agree” and “Strongly Agree” responses. Respondents indicating “Not Applicable” are excluded 
from these results. 
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The findings from the School Visits Study provide additional evidence regarding the extent to 
which the TAs followed the protocols and instructions to administer the assessments as 
intended. The observations of PARCC testing in schools are essential to understanding how well 
TAs followed the test administration protocols and instructions. Observer ratings (excluding Did 
Not Meet or Partially Met) from the school visits indicated that approximately 94% of the test 
administration tasks (39 tasks in total) were completed by TAs as intended across the 109 
observations. The results from the field test study using the same metric indicated that 88% of 
the test administration tasks were completed as intended by TAs. 


Tables 9 and 10 summarize observer ratings for the tasks TAs conduct before students arrive 
and during testing. When reviewing the data in the tables, it is important to remember that 
“Not Observed” was used when it was evident the task occurred but was not directly observed 
(e.g., the testing room was set up well in advance of the observer’s arrival) and “Not 
Applicable” was used when the task was not necessary, such as covering content related 
posters when there were none in the room. 


Table 9 includes tasks completed prior to student arrival in the testing room. There was one 
task (Draw timing box on board) that was either rated “Did Not Meet” or as “Partially Met” in 
48 sessions (44%) across both CBT and PBT administrations. It is important to note that in only 
four of those sessions did observers note that no board was available to use (such as ina 
lunchroom). Observers also noted for the “Partially Met” rating that timing boxes were either 
not updated or the session type or overall time was not listed. These findings are comparable to 
the findings reported in the field test study where observers noted that the timing boxes were 
either not used or were partially used in approximately 53% of the test sessions. Also, for this 
task in PBT administrations the “Not Applicable” rating was used when students were either 
blind or the accommodation was Read Aloud (17%). A second task relevant to PBT only (Post 
example STOP and GO ON signs) was rated as either “Did Not Meet” or as “Partially Met” in 
seven of 18 sessions (39%). Observers in three of the sessions noted that TAs discussed the 
STOP and GO ON signs and students indicated that they knew what the signs looked like. 
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Table 9. Summary of Pre-Assessment Observation Checklist Items 


CBT PBT 
Partially | Did Not Not Partially | Did Not Not 
Met Met Meet Observed NA Met Met Meet Observed NA 
% % % % % % % % % % 
Prepare testing room. 69.2 13.2 0.0 14.3 3.3 100.0 0.0 0.0 0.0 0.0 
Prepare Test Administrator computer. 75.8 17.6 0.0 5.5 1.1 
Prepare student computers. 54.9 6.6 2.2 36.3 0.0 
Ensure prohibited materials not visible. 90.1 0.0 2.2 1.1 6.6 88.9 5.6 0.0 0.0 5.6 
Confirm student roster matches 
Student Authorization Tickets. 67.0 3.3 0.0 29.7 0.0 
Draw timing box on board. 50.6 17.6 26.4 5.5 0.0 38.9 22.2 22.2 0.0 16.7 
Post example STOP and GO ON signs. 55.6 11.1 27.8 0.0 5.6 
Allow authorized visitors only. 91.2 1.1 0.0 0.0 7.7 94.4 0.0 0.0 0.0 5.6 


Note. NA= Not applicable. 
Number of observed CBT sessions is n = 91; number of observed PBT sessions is n = 18. 
Shaded rows indicate items not applicable to that administration type. 
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Table 10 includes tasks completed during and immediately following testing. Nearly all 
applicable tasks were rated as “Met” by observers. There were only a few tasks that received a 
“Did Not Meet” rating. First, for three subtasks under the heading of “Distribute the following 
materials when indicated by the script” observers rated “Did Not Meet” because those 
materials were handed out before the script directed the TAs to do so. Second, the task 
“Provide Seal Code and write Start/Stop times in timing box when script advises” received 
ratings of “Did Not Meet” in four PBT sessions because the TAs failed to update the timing box. 
Third, for the task, “Monitors student progress using administration computer,” observers used 
the “Did Not Meet” rating for the CBT administrations because the administration computer 
was set up outside of the testing room and was monitored by school or district staff (TC or 
technical support staff) other than the TA. The subtasks under the heading of “Handle 
technology problems as trained” was rated “Did not Meet and “Partially Met” in five sessions 
where TAs did not overtly provide additional time to students impacted by the delays. It should 
be noted that no observer indicated that any of those students were unable to finish the test 
because of the delay. Finally, for the task, “Answer student questions following script 
guidance,” the “Did Not Meet” rating was used by observers in four sessions. TAs in three 
sessions at different schools helped the students use the highlight text tool when the students 
could not figure out how to answer that type of test question. In those cases, observers noted 
that the TA provided assistance with the test question. At the fourth session, the TA refused to 
answer any student questions during testing. When asked about this during the debriefing 
interview, the TA indicated that the training she received made it clear that she was not 
supposed to talk to students and they were not supposed to talk to her during testing. 


Also, it is important to note that for this study all but one TA followed the script verbatim or 
very closely. This is in sharp contrast to the observation findings for the field test study where 
observers rated this task as “Did Not Meet” or “Partially Met” in 42 of 77 observations (43%). 
For that study, observer notes indicated that TAs either did not follow the script verbatim or 
took wide liberties when reading the script. When TAs were asked about the script in the 
debriefing interviews, they typically stated that the script was too lengthy and difficult to follow 
when administering the test, which prompted them to make edits to the script during the field 
test administration. 
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Table 10. Summary of Assessment Observation Checklist Items 


CBT PBT 
Partially Did Not Not Partially Did Not Not 
Met Met Meet Observed NA Met Met Meet Observed NA 
% % % % % % % % % % 


Ensure students leave belongings in central 
area. 90.1 2.2 0.0 5.5 2.2 88.9 0.0 0.0 11.1 0.0 
Distribute the following materials when indicated by the script: 


Student Authorization Ticket. 92.3 2.2 0.0 5.5 0.0 
Test Booklet. 83.3 0.0 5.6 11.1 0.0 
Scratch paper and pencil. Print name in 
upper right-hand corner 90.1 4.4 1.1 4.4 0.0 77.8 5.6 5.6 5.6 5.6 
Hand-held calculators 21.9 2.2 0.0 2.2 73.6 11.1 0.0 0.0 0.0 88.9 
Math reference sheet, ruler, compass, or 
protractor 8.8 1.1 1.1 2.2 86.8 27.8 0.0 11.1 0.0 61.1 
Headphones 59.3 3.3 0.0 5.5 31.9 
Ensure students log in and have access to the 
test. 94.5 1.1 0.0 4.4 0.0 
Verify student roster after being seated. 83.3 5.6 0.0 11.1 0.0 
Replace torn or damaged test booklet 0.0 0.0 0.0 0.0 100.0 
Read script verbatim. 86.8 8.8 1.1 3.3 0.0 72.2 22.2 0.0 5.6 0.0 
Provide seal code and write Start/Stop times 
in timing box when script advises. 70.3 25.3 1.1 3.3 0.0 38.9 22.2 22.2 5.6 11.1 


Handle technology problems as trained: 
Individual: adjust remaining time, if not 


timely move/dismiss, resume later 26.4 3.3 3.3 1.1 65.9 
Multiple students: pause testing, update 
timing box with new times 6.6 1.1 1.1 1.1 90.1 
Monitor student progress using 
administration computer. 80.2 0.0 6.6 9.9 3.3 
Monitor students by circulating around the 
room. 100.0 0.0 0.0 0.0 0.0 77.8 0.0 0.0 0.0 22.3 
TA monitors test administration. 83.5 3.3 11.0 0.0 2.2 


(continued) 
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Table 10. Summary of Assessment Observation Checklist Items (continued) 


CBT PBT 
Partially Did Not Not Partially Did Not Not 
Met Met Meet Observed NA Met Met Meet Observed NA 
% % % % % % % % % % 

Handle contaminated (e.g., cut, illness) test 
materials appropriately 0.0 0.0 0.0 0.0 100.0 
Keep track of time accurately. 80.2 7.7 3.3 2.2 6.6 72.2 0.0 5.6 5.6 16.7 
Answer student questions following script 
guidance. 74.7 3.3 1.1 4.4 16.5 38.9 0.0 11.1 11.1 38.9 
Handle student alert to unanswerable or 
misprinted test item. 5.5 0.0 0.0 1.1 93.4 0.0 0.0 0.0 0.0 100.0 
Provide a verbal prompt when 10 minutes 
remain. 62.7 0.0 5.5 5.5 26.3 44.4 0.0 5.6 11.1 38.9 
Provide group a 3 minute break if necessary. 24.1 3.3 2.2 1.1 69.2 44.4 0.0 0.0 0.0 55.6 
Handle testing irregularities. 46.2 0.0 0.0 1.1 52.7 0.0 0.0 0.0 0.0 100.0 
Handle student misconduct appropriately. 16.5 3.3 0.0 1.1 79.1 11.1 0.0 0.0 0.0 88.9 
Handle students finishing early appropriately. 91.2 3.3 0.0 1.1 4.4 72.3 5.6 5.6 0.0 16.7 
Students receive accommodation in regular 
session. 12.1 2.2 0.0 1.1 84.6 11.1 0.0 0.0 0.0 88.9 
Ensure students receiving the same 
accommodations are grouped together. 15.4 0.0 0.0 2.2 82.4 50.0 0.0 0.0 11.1 38.9 
Post-Assessment Items: 

Students log off of the session. 93.4 2.2 0.0 3.3 1.1 

Collect Student Authorization Ticket (CBT) 

or test booklet (PBT) and any test 

materials. 91.2 2.2 0.0 4.4 2.2 88.9 0.0 0.0 11.1 0.0 

Return testing materials to TC 49.5 0.0 0.0 30.7 19.8 44.4 0.0 0.0 33.3 22.2 


Note. NA= Not applicable. 

Number of observed CBT sessions is n = 91; number of observed PBT sessions is n = 18. 
Shaded rows indicate items not applicable to that administration type. 

*Seal code applies to CBT only. 
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In addition to the ratings from the observation checklist presented in Tables 9 and 10, findings 
from the debriefing interviews provide additional contextual information to help explain the 
degree to which TAs followed protocols and instructions. The debriefing interviews reveal that 
in about half of the interviews TAs felt they could find the information they needed in the TA 
manual to administer the operational assessment. 


In summary, training is an important component for ensuring that TAs follow standardized 
procedures for test administration (AERA, APA, NCME, 2014, p. 114), and the findings from the 
TA Surveys indicate that the majority of TAs administering CBTs and PBTs reported that the 
PARCC online trainings, particularly the Introduction to PARCC module, effectively prepared 
them to administer the PARCC assessments. Even higher percentages of TAs reported that the 
training they received from their school effectively prepared them to administer PARCC 
assessments. In addition, most TAs reported that they gained additional familiarity with PARCC 
content via sample items, practice tests, and the PARCC tutorial on TestNav (CBT only) prior to 
test administration. Only about a third of TAs administering PBTs indicated that they reviewed 
tutorials prior to test administration, but of those that completed the PARCC paper-based 
student tutorial, most agreed that it helped better prepare them for administering the 
assessment. These findings help support that the TAs were effectively trained to administer the 
PARCC assessments as intended. Although the TA Survey findings do indicate that training on 
handling technology-related problems could benefit from additional enhancements. Findings 
from the School Visits Study indicate that TAs followed standardized procedures during test 
administration with very few exceptions, which also speaks to the effectiveness of the training 
and training materials. This is in contrast to findings from the field test study where, in nearly 
half of the sessions observed, TAs did not follow the script in the TA Manual verbatim. The 
improvements observed in TAs following the script in Year 1 are likely partially attributed to the 
actions PARCC took since the field test to improve the clarity of the TA Manual and to improve 
the effectiveness of the TA training, including the addition of more opportunity to interact with 
PARCC content prior to test administration. Consequently, greater adherence to the 
standardized test administration procedures, through improved training, additional exposure to 
PARCC content prior to test administration, and improved clarity of the TA manual, appears to 
have helped strengthen adherence to Standard 6.1 since the field test. 


Findings for Research Question 2 Specific to Accessibility Features and Accommodations 


The TA Surveys also specifically asked TAs about the effectiveness of the training materials 
related to administering Accessibility Features and Accommodations (AFAs). Both the CBT and 
the PBT TA Surveys included the same set of questions on AFAs. To provide some context for 
interpreting the findings on the usefulness of the AFA training materials, Table 11 reports the 
percentages of TAs who indicated that they administered each type of AFA. Table 11 shows that 
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“Extended Time” was the most frequently administered AFA with approximately a third of TAs 
indicating that they administered this AFA. The next most frequently administered AFA was a 
Human Reader for Mathematics on the PBTs (both PBA and EOY) with approximately a fourth of 
the TAs administering PBTs indicating that they implemented this AFA. The Human Signer and 
the External Word Prediction Device were the most infrequently administered AFAs with less 
than 1% of all TAs indicating that they administered these AFAs. 


Table 11. Percentage of TAs Indicating They Administered Particular AFAs 


Accessibility & Accommodation % CBT PBA % CBT EOY % PBT PBA % PBT EOY 

Feature (N=10,545) (N= 11,232) (N = 2,336) (N = 1,469) 
Human reader for mathematics 7.3 8.1 27.0 24.9 
Human reader for ELA 4.3 4.9 15.2 15.6 
Human scribe 4.7 4.3 10.1 10.7 
Human signer 0.3 0.2 0.5 0.3 
menos read aloud in native 16 40 57 55 
External speech-to-text device 6.0 6.5 1.0 0.8 
Extended time 31.2 32.6 38.2 36.6 
External word prediction device 0.7 0.7 0.4 0.7 
Calculator on non-calculator section 12.6 16.2 15.2 16.5 
Other external device 1.6 1.3 3.0 2.50 


Note. Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 

The TA Surveys also asked TAs to indicate if they had read the approved PARCC Accessibility 
Features and Accommodations Manual prior to test administration. The results show that in 
Year 1 over 80% of the TAs indicated that they read the manual, whereas only about two-thirds 
of TAs indicated that they read the manual prior to administering the field test (see Figure 12), 
a notable increase. 
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TA Survey: "Did you read the PARCC Accessibility Features 
and Accommodations Manual prior to administration?" 
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Figure 12. Percentage of TAs indicating they read the PARCC Accommodations Manual. 
Percentages based on valid (non-missing) responses. 


The TAs were also asked to rate the extent to which they agreed that a particular AFA training 
component effectively informed them of how to administer AFAs on the PARCC assessments. 
Those results are presented in Figure 13. Of those indicating the AFA training component was 
applicable, the majority of TAs across test formats (CBT and PBT) and across test types (PBA and 
EOY) agreed that the training item/component effectively informed them of how to administer 
AFAs on the PARCC assessments. School training and the AFA Manual received the highest 
percentage of agreement ratings (80% - 88%) and state DOE training received the lowest 
percentage of agreement ratings (55% - 64%). 


Next, the TAs were asked to indicate the extent to which they agreed that a particular appendix 
from the TA Manual or a particular guideline effectively informed them of how to administer 
AFAs. Those findings are presented in Figure 14. Of those for whom the appendix/guideline was 
applicable, more than three-fourths agreed that it effectively informed them of how to 
administer AFAs. This result was found for all appendices/guidelines and across both test 
formats (CBT and PBT) and across both test types (PBA and EOY). 
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TA Survey: "This particular item/component effectively informed me of how to m CBT PBA 
administer accessibility features and accommodations on the PARCC assessments. m CBT EOY 
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Figure 13. Percentage of TAs agreeing that a specific manual or training informed them of how to administer accessibility features and accommodations. 


Percentages based on valid responses. Results reflect the combined percentage of “Agree” and “Strongly Agree” responses. Respondents indicating “Not 
Applicable” are excluded from these results. For this set of items, n counts for “applicable” respondents ranged from n = 6,673 to 9,651 for CBT PBA; from 
n = 7,253 to 10,301 for CBT EOY; from n = 1,360 to 2,077 for PBT PBA, and from n = 885 to 1,336 for PBT EOY. 
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TA Survey: "This particular appendix/guideline effectively informed me of how to 
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Figure 14. Percentage of TAs agreeing that a specific appendix or guideline informed them of how to administer accessibility features and accommodations. 
Percentages based on valid responses. Results reflect the combined percentage of “Agree” and “Strongly Agree” responses. Respondents indicating “Not Applicable” are 
excluded from these results. Across this set of items, n counts for “applicable” respondents (including missing responses) ranged from n = 3,552 to 5,523 for CBT PBA; from n= 
3,843 to 6,020 for CBT EOY; from n = 789 to 1,597 for PBT PBA, and from n = 522 to 972 for PBT EOY. 

Appendix A = Accessibility Features and Accommodations for Students taking the Paper-Based PARCC Assessments; Appendix B = Test Administration Protocol for the Human 
Reader Accommodations for ELA/L and the Human Reader Accessibility Feature for Math; Appendix C =Protocol for the Use of the Scribe and for Transcribing Student 
Responses; Appendix E = Guidance for Selecting and Administering the Extended Time Accommodation; Appendix | = PARCC ELA Audio Guidelines; Appendix J = PARCC Math 
Audio Guidelines. 
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Three additional TA Survey questions pertaining to AFA trainings and materials were also asked 
of TAs. These final three AFA questions were specific to the Personal Needs Profile (PNP). First, 
TAs were asked whether the process for identifying AFAs in advance via the PNP were clear and 
easy to follow. Second, as a follow up, TAs were asked whether the AFAs pre-identified in the 
PNP were made available to students during a practice session with sample items, tutorials, or a 
practice test. Finally, TAs were asked to indicate whether changes were made to a student’s 
PNP or to the availability of accommodations or accessibility features during test 
administration. For those for whom the PNP was applicable, approximately two-thirds of TAs 
across test formats (CBT and PBT) and across test types (PBA and EOY) agreed that the process 
for identifying AFAs via the PNP was clear and easy to follow (see Figure 15). Only slightly more 
than a third, however, indicated that the AFAs pre-identified via the PNP were made available 
to students during a practice session; the majority of TAs did not know whether the AFAs were 
made available to students during a practice session (see Figure 16). Finally, very few TAs (5% - 
7%) indicated that changes were made to a student’s PNP or to the availability of AFAs during 
test administration; many were unaware as to whether such changes were made (see 

Figure 16). 


TA Survey: "The process for identifying accessibility features 
and accommodations in advance (via the Personal Needs 
Profile and associated trainings) was clear and easy to follow." 
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90.0% 
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Figure 15. Percentage of TAs agreeing that the process for identifying accessibility features and 
accommodations was easy to follow. 


Percentages based on valid responses. Results reflect the combined percentage of “Agree” and “Strongly Agree” 
responses. For the values presented here, “Not Applicable” responses were excluded. 
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TA Survey: Availability of Accessability Features & Accomodations 
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Were all AFAs that were pre-identified in the PNP made | During administration, were changes made to a student's 
available to students during a practice session? PNP or to the availability of AFAs? 


Figure 16. Percentage of TAs indicating availability of AFAs. 
Percentages based on valid (non-missing) responses. For the values presented here, “Not Applicable” responses were excluded. 
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During the school visits, HumRRO observed 22 accommodation sessions which often included 
multiple accommodations. The accommodations observed included: 3 Read Aloud to 
themselves, 4 Human Reader, 8 Text-to-Speech, 15 Extended Time, and 14 Other (e.g., scribe, 
Braille, large print, note taker). The findings from these sessions indicate that no observed 
session received a rating of “Did Not Meet” and that only two sessions (9%) had a “Partially 
Met” rating because the students with an extended time accommodation were included in the 
same room with students receiving no accommodation; however, in both cases students 
finished within the regular testing time. Observer notes from six observed sessions (out of the 
22 observed accommodation sessions) indicated that additional training or prompts in the 
script may be useful because there was some confusion with adjusting headphone volume (it 
should be noted that PARCC is already addressing this), needing new seal codes when 
continuing to another session (it should be noted that PARCC is eliminating seal codes), and 
whether or not human scribes should flip through the test booklets (PBTs) to ensure all 
questions are answered. 


Additional information regarding accommodations was received during 15 interviews. In eight 
of the 15 interviews, TAs indicated that there were no problems with administering 
accommodations to their students. With regard to training, TAs in two interviews reported that 
they only reviewed the manual, and in six interviews TAs indicated they received more 
extensive training from their district or state. There were four interviews where TAs reported 
some problems, such as the text-to-speech or magnification did not work on some questions or 
that they were confused by differences between PARCC and state accommodations. Some TAs 
suggested that students should be able to tailor the text reading option to what is read 
(stimulus, answer options, passages), adjust the volume during testing, and adjust the speed of 
the text reading (e.g., slow, medium, fast). (/t should be noted that PARCC has already 
addressed the speed of text reading issue, and students can now adjust the speed of text 
reading). 


In summary, the findings related to accessibility features and accommodations indicate that the 
vast majority of TAs reported reading the AFA Manual prior to test administration; this was a 
notable increase from the field test. Moreover, the majority of TAs agreed that the AFA 
trainings and the AFA guidelines effectively prepared them to administer AFAs. Observer ratings 
from the observed accommodation sessions also indicate that in nearly all of the observed 
accommodation sessions the TAs performed the tasks related to accommodation sessions as 
intended. One area in which improvements might be made are for the PNP training and for the 
process for identifying AFAs in advance via the PNP. The TA Survey results suggest that 
improvements could be made to the process for identifying AFAs in advance via the Personal 
Needs Profile (PNP) so that it is clearer and easier to follow. Moreover, the survey findings 
indicate that the majority of TAs responded either “no” or “not sure” when asked whether all 
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the AFAs identified in the PNP were made available to students during a practice session. This 
finding suggests that PARCC leadership may want to incorporate some additional scrutiny to 
ensure that students who need AFAs are indeed given the opportunity to practice with those 
AFAs prior to actual test administration (it should be noted that PARCC now has several paper- 
based and online practice tests available for the 2015 Fall Block administration®). Efforts to 
improve the PNP-related training/components and the clarity of the process for identifying 
AFAs via the PNP should help to strengthen adherence to Standard 6.2. 


Findings for Research Question 3: Do the students appear to understand the instructions 
provided to them? 


The claim in the theory of action most directly related to Research Question 3 is, “Students 
respond to items as intended” (claim 4). For this claim to be true, important assumptions that 
must be verified are that: (a) students understand the directions provided by the TA, and 

(b) students understand the directions on the test. Findings from all three studies help to 
inform this research question and its underlying assumptions.” 


Regarding whether students understand the directions provided by the TAs, the Student 
Surveys (both the mathematics and the ELA versions) ask students to indicate, “Did you 
understand all of the directions read by the person who gave you the test?” The Student Survey 
results show that the vast majority of students (approximately 80% or more) understood the 
directions read by the TA; this was true for both the field test and for Year 1 (see Figure 17). The 
TA Surveys (both the CBT and PBT versions) also addressed the question of whether students 
understood the directions read to them by the TAs. The TAs were asked to indicate their 
agreement with the statements, “Students appeared to understand the instructions | read to 
them during test administration,” and “The instructions | read to the students covered all of the 
information necessary to take the test.” The results presented in Figure 18 indicate that the 
majority of TAs administering the CBTs (85% — 88%) and the majority of TAs administering the 
PBTs (76% - 83%) agreed that students appeared to understand the instructions they read to 
them. The results presented in Figure 19 indicate that the majority of TAs administering CBTs 
(76% - 80%) and PBTs (72% - 79%) agreed that the instructions they read to students covered all 


8As of 9-11-15, PARCC now has practice tests available for: 
e Paper (regular) 
o. Large print 
o Spanish 
o Spanish large print 
o Braille 
e Online (regular) 
o Spanish 
o Text-to-speech 
o Spanish Text-to-speech 
oO Screen reader 
o American Sign Language 
°Additional evidence for Research Question 3 is presented under Research Question 7b. Research Question 7b 
asks, “What types of questions, if any, do students ask during the test administration?” 
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the information needed for students to take the test. For both of these TA Survey questions, 
the results for Year 1 represent a notable increase from the field test, particularly for TAs 
administering PBTs. 


Student Survey: "Did you understand all of the directions 
read by the person who gave you the test?" 
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Figure 17. Percentage of students indicating they understood all the directions read by the TA. 
Percentages are based on the number of valid (non-missing) responses. 


Submitted September 28, 2015 Page 41 


PARCC Quality of Test Administration 


TA Survey: "Students appeared to understand the directions 
| read to them during the test administration." 
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Figure 18. Percentage of TAs agreeing that students appeared to understand the directions they read 
to them. 


Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect 
the combined percentage of “Agree” and “Strongly Agree” responses. 


TA Survey: The instructions | read to the students covered 
all of the information necessary to take the test. 


m@ CBT PBA 
100.0% N=10,545 
90.0% m CBT EOY 
80.0% N = 11,232 
70.0% m PBT PBA 
N = 2,336 
a 60.0% 
= m PBT EOY 
Pa N= 1,469 
XS 40.0% : 
@ Field Test CBT 
30.0% n= 4,635 
20.0% m Field Test PBT 
10.0% n= 1,687 
0.0% 


Figure 19. Percentage of TAs agreeing that the instructions they read to students covered all of the 
information students needed to take the test. 


Percentages based on valid (non-missing) responses. Ratings made on a 4-point agreement scale. Results reflect 
the combined percentage of “Agree” and “Strongly Agree” responses. 
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Next, with regard to students’ understanding of the directions for the questions on the test, the 
Student Survey results indicate that the majority of students (approximately half) reported that 
it was hard to understand the directions for the questions on the test “some of the time.” As 
seen in Figure 20, this was true for both the field test and for the first year of the operational 
test. Findings were similar for Mathematics and ELA, although there is a slight trend indicating 
that the directions on the Mathematics tests might be somewhat more difficult to understand, 
as evidenced by the slightly higher percentage of students indicating that it was hard to 
understand the directions on the mathematics tests “almost always” or “most of the time” on 
both the field test and the operational test. These results indicate a lack of support for the 
assumption that students understand the directions for the questions on the test; most students 
(roughly 75% across subjects and administration modes) indicate that it was hard to understand 
the directions for the questions on the test at least some of the time. If students are having 
difficulty understanding the directions on the test, then they may not be responding to the test 
items as intended, which threatens the veracity of Claim 4 from the TOA (i.e., Students respond 
to items as intended). 


Student Survey: "How often was it hard to understand 


the directions for the questions on the test?" 
(Year 1 Results Compared to Field Test Results for Math and ELA) 
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Figure 20. Percentage of students indicating that it was hard to understand the directions for the 
questions on the test. 
Percentages are based on the number of valid (non-missing) responses. 
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A factor that may affect whether students understood: (a) the instructions provided to them by 
the TA and (b) the directions on the test for responding to test items, is whether the students 
had an opportunity to gain familiarity with the PARCC assessments prior to actual 
administration. Given that the PARCC assessments are new to students, it is reasonable to 
expect that students may be better positioned to understand the instructions if they had 
already practiced with PARCC content prior to test administration. This is consistent with the 
guidance for Standards 4.16 and 6.5. Consequently, the TA Surveys asked TAs to indicate the 
various ways that students in their session practiced with PARCC content prior to 
administration and the Student Surveys asked students, “How many times did you practice ona 
computer or tablet to get ready for this test?” 2° 


Results from the TA Surveys indicate that approximately two-thirds of TAs administering CBTs 
indicated that students in their session(s) completed sample items and the practice test prior to 
test administration, and about half indicated their students completed the tutorial prior to test 
administration (see Figure 21). There was a consistent trend of more practice occurring for 
students taking the CBT than for students taking the PBT. This was particularly true for practice 
with the tutorials with over 50% of TAs administering CBTs indicating that their students 
practiced with the tutorials, whereas less than 15% of TAs administering PBTs indicated that 
their students practiced with the tutorials (see Figure 21). 


TA Survey: "How did students in your session(s) practice with 
PARCC content prior to administration?" (Select all that apply) 
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Figure 21. Percentage of TAs indicating how students in their session practiced with PARCC content. 


Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. Comparisons with the field test are not 


Recall that the Student Survey was only administered to students taking the EOY CBTs (math and ELA). The 
Student Survey was not administered to students taking PBTs. 


Submitted September 28, 2015 Page 44 


PARCC Quality of Test Administration 


presented due to differences in content of the survey questions and differences in the format of the survey 
questions from the field test to Year 1. 

The Student Survey results also shed some light on students’ interaction with PARCC content 
prior to test administration. The results in Figure 22 indicate that the majority of students 
taking the CBTs practiced one or more times on a computer/tablet prior to taking the test. Less 
than a third of the students reported that they “never” practiced on a computer/tablet to get 
ready for the operational test. This coincides with the results from the TA Survey, which 
indicated that roughly two-thirds of TAs reported that students taking CBTs completed sample 
items and took the practice test. Figure 22 also shows that the percentage of students reporting 
that they practiced on a computer/tablet “more than once” nearly doubled from the field test 
to Year 1 (from about 20% to about 40%). 


Student Survey: "How many times did you practice 


on a computer or tablet to get ready for the test?" 
(Year 1 Results Compared to Field Test Results for Math and ELA) 
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Figure 22. Percentage of students indicating how many times they practiced for CBT. 
Percentages are based on the number of valid (non-missing) responses. 


Finally, findings from the observation study also shed some light on Research Question 3. While 
there were no direct “tasks” on the observation checklist associated with students 
understanding the instructions, the observers did provide some insight into the types of 
questions students asked during the test. Student questions that observers noted from 20 CBT 
sessions (out of 91) were fairly evenly distributed across the following topics related to the use 
of technology: tool usage in order to answer questions, navigation related questions (e.g., 
scrolling), and logging on or logging off questions. Other questions from students that were not 
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related to the use of technology included: requesting help with unfamiliar words, diagrams, 
and other content-related questions to which the standard TA reply was, “...do your best.” 


In summary, most students reported that they understood the directions read to them by the 
TA. The majority of TAs also reported that students appeared to understand the directions they 
read to them and that the instructions covered all of the information necessary for students to 
take the test (for CBT and PBT); the percentages of TAs agreeing with these statements 
increased notably from the field test. Consequently, these results indicate that students 
understood the instructions provided by the TAs. On the other hand, the findings indicate that 
students had more difficulty understanding the directions on the test for responding to test 
items. Most students taking the CBT assessments reported that it was hard to understand the 
directions for the questions on the test at least some of the time. And findings from the 
observation study indicate that when students asked questions they were most likely to ask 
questions about how to use the technology within the assessment. Per Standards 4.16 and 6.5, 
students are more likely to understand test instructions if they have had practice with the items 
and with the mode of testing (e.g., computer-administered) prior to actual testing. Overall, the 
findings indicate that most students reported practicing on a computer/tablet one or more 
times prior to actual testing, which was an increase from the field test. Relatedly, the TAs 
administering CBTs indicated that almost two-thirds of the students practiced with sample 
items and a practice test prior to actual testing. However, only about half of the students 
completed the tutorial prior to taking the computer-based assessment (as reported by TAs). If 
more students taking the CBTs completed the tutorials prior to test administration, then this 
might help to improve students’ understanding of the directions on the test and item types, 
particularly with regard to using the technology embedded in the assessment. With regard to 
the PBTs, the TA Survey results indicate that very few students completed the tutorial directed 
towards paper-based assessments. Consequently, additional efforts should be made to 
encourage completion of the student tutorials for PBT administration. In sum, the evidence 
indicates that students understand the instructions read by the TAs, but that they have more 
difficulty understanding the directions on the test for responding to test items. Additional 
clarity and detail in the instructions on the test and increased practice opportunities via 
completion of student tutorials are two efforts that could help tighten adherence to Standards 
4.16 and 6.5. 


Findings for Research Question 4: To what degree are students engaged in taking the test? 


Research Question 4 also most directly addresses the claim that, “Students respond to items as 
intended” (claim 4). For this claim to be met, students must have some engagement with the 
assessment. Consequently, another assumption underlying the claim “that students respond to 
items as intended” is that students are engaged in taking the test. If students are not engaged 
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or if they have low levels of engagement, then they might not sufficiently attend to the items so 
as to respond to the items as intended. 


There are many factors that can influence student engagement—for example, distractions from 
outside visitors, problems with the technology in the assessment, too little time to complete 
the assessment, to name just a few. The impacts of several of these types of potential factors 
were investigated through the three studies. First, from the School Visits Study, the observer 
ratings provided an indication of the extent to which students may have been distracted by 
outside visitors. Ratings from the observation checklist indicate that of the 109 observed 
sessions, all observations except one were either rated “Met” or “Not Applicable” for the task 
“no unauthorized visitors permitted during the testing session.” The one exception was 
lunchroom staff preparing for student lunches for the first 15 minutes while students tested in 
the lunchroom; this may have been a source of distraction for these students and thus an 
impediment to student engagement. 


Another possible indication of student engagement is the amount of time students took to 
complete the test. If students are rushing through the test and/or skipping items to finish on 
time, then student engagement is negatively impacted, which means students likely are not 
responding to items as intended. Conversely, if students are finishing the test very early, then 
that might indicate that students are not taking the time to carefully read the instructions and 
the test items in order to respond to the items to the best of their ability. Ideally, we would 
want to see the majority of students finishing the test on time. The results in Figure 23 show 
that the majority of students completing the CBTs did indeed indicate that they finished the 
test on time (48% - 49%), although there were nearly as many students indicating that they 
finished the test very early (44% - 45%). Figure 23 also shows that the majority of TAs (56%) 
reported that students finished the CBTs very early, although a sizeable percentage (42%) of 
TAs reported that the students finished the CBTs on time. These findings indicate that the 
length of time allotted for the CBTs is not so short that students must rush through the test to 
finish on time. However, given the sizeable percentage of students and TAs indicating that 
students “Finished Very Early,” we cannot rule out the possibility that many students may not 
have taken their time to carefully read and respond to all the items on the test to the best of 
their ability. Another likely explanation for the high percentage of “Finished Very Early” is that 
there may have been too much time allotted for the CBTs. Finally, while there was no Student 
Survey administered to students taking the PBTs, results from the PBT version of the TA Survey 
show that the majority of TAs administering PBTs indicated that students finished the PBTs “On 
Time” (62% for the PBT PBA and 52% for the PBT EOY). 
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Did students have enough time to finish the test? 


(Comparison between Student and TA Responses) 
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Figure 23. Percentages of students and TAs indicating whether students had enough time to finish the 


test (CBT). 
Percentages are based on valid (non-missing) responses. 


*Results from the TA Survey for CBT EOY administration are presented here to compare with results from the same 
assessment (CBT EOY) on which students provided survey responses. Results from the TA Survey for CBT PBA 
administration were similar to the results for CBT EOY administration. No comparison with the field test results for 
the TA Survey are provided because a different question was asked on the field test TA Survey. 

Another factor that may impact student engagement is students’ familiarity with the content 
on which they are being tested. If most of the test questions cover content that is unfamiliar to 
the students, then students might “give up” on the test. Relatedly, if most of the test questions 
are more difficult than the content to which students are accustomed, then students might also 
be more likely to “give up.” To help inform whether these might have been concerns for the 
PARCC assessments, the students were asked to indicate, “How many questions asked you 
about things you have not learned in school this year?” and “How difficult was this test?” Less 
than a third of the students indicated that “most” or “all” of the questions on the mathematics 


assessments asked them about things they have not learned, and less than a fourth of the 
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students indicated that “most” or “all” of the questions on the ELA assessments asked them 
about things they have not learned (see Figure 24). The majority of students (approximately 
half) indicated that “a few” of the questions asked them about things they have not learned in 
school (for both mathematics and ELA). These survey results were very similar to the results 
from the field test (see Figure 24). With regard to the difficulty of the test, the majority of 
students (51%) indicated that the mathematics assessment was “harder than their school 
work,” although the percentage of students endorsing this response option was somewhat 
lower than it was for the field test (51% as compared to 59%) (see Figure 25). For the ELA 
assessment, the majority of students (51%) indicated that the difficulty of the ELA test was “the 
same as their school work.” This result was similar to the result from the field test (see 

Figure 25). Overall, the set of findings depicted in Figure 24 indicate that the majority of 
students were familiar with most of the content on the assessments. Consequently, lack of 
familiarity with test content was not likely to have been a major impediment to student 
engagement for the majority of the test takers. On the other hand, there is some evidence to 
suggest that students might have been dissuaded by the difficulty of the mathematics 
assessment (Figure 25), which could have negatively impacted student engagement, although 
this result could also be an indication that students’ mathematics school work is not sufficiently 
challenging. These findings should be taken into consideration in conjunction with other 
findings from the studies. 


Submitted September 28, 2015 Page 49 


PARCC Quality of Test Administration 


Student Survey: "How many questions asked you 


about things you have not learned in school this year?" 

(Year 1 Results Compared to Field Test Results for Math and ELA) 
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Figure 24. Percentage of students indicating how many questions asked about things not learned in 
school. 
Percentages are based on valid (non-missing) responses. 
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Student Survey: "How difficult was this test?" 
(Year 1 Results Compared to Field Test Results for Math and ELA) 
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Figure 25. Percentage of students indicating test was harder, same, or easier than school work. 
Percentages are based on valid (non-missing) responses. 


A factor that might impact student engagement that is specific to the computer/tablet-based 
assessments is students’ familiarity with using computers/tablets. If students do not have the 
opportunity to work with computers/tablets at home or at school, then this mode of testing 
would be unfamiliar to them. The findings presented in Figures 26 and 27 indicate that lack of 
familiarity with computers/tablets is not likely to have been an impediment to student 
engagement for the vast majority of students. More than half of the students indicated that 
they use a computer or tablet every day at home and only 7% indicated that they do not use a 
computer/tablet at home. Moreover, two-thirds or more indicated that they use a computer or 
tablet at school a few times a week or more, and only 3% indicated that they do not use a 
computer or tablet at school. These results were very similar to the results from the field test. 
Consequently, based on these Student Survey results, the student test-takers are very familiar 
with using computers/tablets. Given this, it is perhaps not surprising that when asked “Would 
you rather take this test on paper or on a computer or tablet?” more than two-thirds indicated 
that they would rather take the test on a computer/tablet (see Figure 28). These findings are 
consistent with findings obtained from the debriefing interviews with TAs; during the debriefing 
interviews some TAs commented that students appeared to enjoy the videos and the different 
types of items (e.g., drag and drop). 
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Student Survey: Use of Computer/Tablet at Home 


and at School 
(Year 1 Results Compared to Field Test Results for Math) 
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Figure 26. Percentages of students indicating how often they use a computer/tablet at home and at 
school (mathematics). 
Percentages are based on valid (non-missing) responses. 
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Student Survey: Use of Computer/Tablet at 
Home and at School 
(Year 1 Results Compared to Field Test Results for ELA) 
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Figure 27. Percentages of students indicating how often they use a computer/tablet at home and at 
school (ELA). 
Percentages are based on valid (non-missing) responses. 
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Student Survey: "Would you rather take this test 


on paper OR on a computer or tablet?" 
(Year 1 Results Compared to Field Test Results for Math and ELA) 


100.0% 
90.0% 
80.0% 
70.0% 
60.0% 
50.0% 
40.0% 
30.0% 
20.0% 
10.0% 

0.0% 


@ Ona Computer or Tablet 


@ On Paper 


CBT EOY Math CBT EOY Math CBT EOY ELA CBT EOY ELA 
Year 1 Field Test Year 1 Field Test 
n = 3,563,623 n= 157,872 n = 3,503,474 n= 160,820 


Math ELA 


Figure 28. Percentage of students indicating they would rather take the test on a computer/tablet. 
Percentages are based on valid (non-missing) responses. 


A final factor explored in these studies that may negatively impact student engagement is the 
occurrence of technology-related problems during the CBT administration. There is 
documented evidence that technology-related problems—of a local nature as opposed toa 
system-wide nature—were a concern for the field test administration (see PARCC Field Test: 
Lessons Learned, PARCC, 2015 and Sinclair et al., 2015). As a result, the PARCC consortium 
implemented several enhancements to help improve the “local technology experience.” Those 
enhancements included additional infrastructure trials prior to test administration, enhanced 
guidance and documentation for the SystemCheck tool, updated training modules on 
conducting infrastructure trials and proctor caching, a new technology preparedness training 
module, and enhanced communication to states and districts regarding technology 
preparedness (see PARCC Field Test: Lessons Learned, PARCC, 2015). For the field test, between 
40% (PBA Math) to 59% (EOY ELA) of students reported that “no technology-related problems 
occurred” during CBT administration. For Year 1, the percentage of students reporting that “no 
technology-related problems occurred” increased to nearly 80% for both the Mathematics and 
ELA CBTs. This indicates that technology-related problems were not an impediment to student 
engagement for the majority of student test-takers, and it also suggests that the consortium’s 
actions taken after the field test to help improve the local technology experience were 
beneficial. Converging findings from the School Visits Study indicate that observers noted that 
technology-related problems occurred in only 27% of the observed sessions (29 of 109 
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sessions), and that the problems were reported to be resolved fairly quickly. This is in contrast 
to the extensive connectivity problems observed during the field test study where 
approximately one-third of the schools visited experienced problems considered severe that 
either took an hour or more to resolve or suspended testing altogether. (Additional information 
and discussion of technology-related concerns are discussed under the results for Research 
Question 7a, “Are there any technology-related problems during test administration?”) 


A final source of evidence on student engagement comes from the School Visits Study. After 
observing test administration, the observers interviewed the TAs and asked them, “Was 
student engagement high, moderate, or low? This question was answered in 67 of the 
interviews, with TAs from 52 interviews (78%) reporting that student engagement was 
moderate or high. Overall, the TAs administering the CBTs indicated that students appeared to 
enjoy the videos and the different types of items (e.g., drag and drop) and that several students 
reported to their TA that they liked the computer test better than the traditional paper and 
pencil test, even though many thought it was more difficult. These types of comments from the 
TAs during the debriefing interviews were consistent with the observer notes in the observation 
checklists. The observers generally noted that students appeared to be engaged during testing 
and were respectful of others who were still working. Typical observer notes included, 
“students remained engaged in spite of delays with connectivity issues,” and “students seem 
focused and motivated.” There were only two schools where observers noted that students 
appeared bored and finished the test in less than 15 minutes (e.g., “students not engaged with 
one asleep and others not following directions”). 


In summary, with regard to Research Question 4, “To what degree are students engaged in 
taking the test?” the evidence from these studies indicate that there was very little evidence of 
distractions caused by unauthorized visitors, students did not have to rush to finish the test, 
most of the questions on the test covered topics students had already learned about in school, 
most students were very familiar with the mode of assessment (CBT) and they preferred that 
mode of assessment, and there were few reports by students and observers of distractions 
caused by technology-related problems. Moreover, feedback obtained from the TAs in the 
debriefing interviews and comments made by observers on the observation checklist indicated 
moderate to high levels of student engagement. Collectively, these findings indicate a 
reasonable degree of student engagement such that a lack of student engagement is not likely 
to have resulted in students not responding to items as intended. Only test difficulty emerged 
as a potential factor that might have negatively impacted student engagement, particularly for 
the mathematics assessments. Most students reported that the mathematics assessments were 
harder than their school work. The primary factors that emerged in the field test as 
impediments to student engagement— overly lengthy and redundant test instructions read by 
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the TAs and delays caused by technology-related problems—appeared to have been reconciled 
since the field test, as these did not emerge as impediments to student engagement in Year 1. 


Findings for Research Question 5: Is there any disruptive student behavior during the session? 


This research question also gets to the heart of the claim that “students respond to items as 
intended” (claim 4). Another assumption underlying this claim is that students’ attention is not 
adversely impacted by other disruptive students. To the extent that disruptive student behavior 
is minimized, then this increases the likelihood that students attend to the test and respond to 
items as intended. 


The observer ratings from the Observation Study provide an indication of the extent to which 
disruptive student behavior was minimized. Ratings from the observation checklist indicate 
that, of the 109 observed sessions nearly all were rated as having ‘Met’ the tasks of: “Ensure 
students leave belongings in central area (i.e., cell phones, other electronics, and reference 
books),” “Monitor students by circulating around the room,” “Handle student misconduct 
appropriately (e.g., talking, cheating, using electronic devices),” and “Handle students finishing 
early appropriately (i.e., following state guidance).” This information helps support the 
assumption that students’ attention was not adversely impacted by disruptive student 
behavior. While disruptive student behavior does not appear to be a widespread concern, 
observers noted there were some instances of students talking or giggling too much in four 
different schools; however, the observers noted that the TAs quickly settled those disruptions. 
There were two schools were a student was dismissed from the test because he or she did not 
quiet down as requested; the observers indicated that the TAs’ actions were appropriate. 


, 


In summary, the findings for Research Question 5 indicate that disruptions caused by students 
behavior during the testing session were minimal. Consequently, there is no need for 
recommendations for improvement in this area. 


Findings for Research Question 6: Are there any apparent attempts to record or copy test 
materials including the test questions by students or others? 


This research question most directly addresses the claim that, “TAs are prepared to administer 
the assessments as intended” (claim 1). An important component of test administration is 
ensuring that the test content remains secure. To do otherwise would threaten the validity of 
the testing system. A source of evidence for this research question is the observations 
conducted as part of the School Visit Study. Several tasks on the observation checklist pertain 
to issues related to test security. The collective set of selected tasks and the degree to which 
they were met (as determined by the trained observers) are presented in Table 12. The results 
in Table 12 indicate that most of the tasks related to test security were rated as “Met.” Tasks 
that did not have a high percentage of “Met” ratings typically occurred because that task was 
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either “Not Observed” or was “Not Applicable” (such as not providing breaks or materials not 
returned to the TC since the TC administered the test). An exception to the “Met” ratings 
occurred for “prepare testing room” for 11 of 109 observations (10%). Observer notes indicate 
that the primary issue with room preparation was with the lack of use of partitions or lack of 
sufficient space between computers as indicated in the manual. Typical observer comments 
were: “seating was tight (not much space)” and “students next to one another, but seemed 
used to it.” Observers did report that a few students appeared to be looking at others’ 
computer screens for the CBT; however, they indicated that those students had finished testing 
and appeared to be bored waiting for others to finish. Also, one observer gave a rating of “did 
not meet” in two sessions for the “Provide group a 3 minute break if necessary” task with a 
note that one student did not log off his or her computer. 
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Table 12. Summary of Observation Checklist Items Pertinent to Test Security 


Ratings 
CBT PBT 
Partially Did Not Not Partially Did Not Not 
Met Met Meet Observed NA Met Met Meet Observed NA 
% % % % % % % % % % 

Prepare testing room. 69.2 13.2 0.0 14.3 3.3 100.0 0.0 0.0 0.0 0.0 
Confirm student roster matches Student 67.0 33 0.0 297 0.0 
Authorization Tickets. 
Allow authorized visitors only. 91.2 1.1 0.0 0.0 7.7 94.4 0.0 0.0 0.0 5.6 
a SHEenES Ir avOSPMRINESIMCenta! baa a2 0.0 5.5 og 88.9 0.0 0.0 11 0.0 
eel SHHCEnS Py CRUIAAEAFOUNE TNE aaa 0.0 0.0 0.0 0.0 77.8 0.0 0.0 0.0 22.4 
Provide group a 3 minute break if necessary. 24.1 3.3 2.2 1.1 69.2 44.4 0.0 0.0 0.0 55.6 
Handle testing irregularities (fire drill) 46.2 0.0 0.0 1.1 52.7 0.0 0.0 0.0 0.0 100.0 
Handle student misconduct appropriately. 16.5 3.3 0.0 14 79.1 11.1 0.0 0.0 0.0 88.9 
Collect Student Authorization Ticket (CBT) or 912 22 0.0 AA 22 38.9 0.0 0.0 1431 0.0 


test booklet (PBT) and any test materials. 
Return testing materials to TC 49.5 0.0 0.0 30.7 19.8 44.4 0.0 0.0 33.3 22.2 


Note. NA= Not applicable. 
Number of observed CBT sessions is n = 91; number of observed PBT sessions is n = 18. 
Shaded rows indicate items not applicable to that administration type. 
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In summary, there was no evidence of overt attempts to record or copy test materials by 
students or others. Findings from the observations of test administration indicate that most of 
the tasks related to test security were rated as “Met,” thereby indicating evidence to support 
adherence to Standards 6.6 and 6.7. An exception to the “Met” ratings occurred for “prepare 
testing room.” Observer notes indicate that the primary issue with room preparation was with 
the lack of use of partitions and the lack of sufficient space between computers. 


Findings for Research Question 7a: Are there any technology-related problems during test 
administration? 


This research question is most closely related to the claim that, “Technology improves and 
facilitates the assessment experience” (claim 3). If interruptions are caused by problems with 
technology, then the veracity of this claim is threatened. All three studies address this research 
question for the CBT administrations. 


First, the results from the TA Survey are presented in Figure 29 with comparisons provided for 
Year 1 and the field test, where applicable. The two most frequently reported problems with 
technology in Year 1 were with logging students onto the assessment and with the system 
disconnecting/logging students out. Over half of the TAs reported that these problems occurred 
in Year 1. With regard to comparisons with the field test results, the findings for Year 1 are very 
similar. For example, for the field test, 13% of TAs reported that “no technology-related 
problems occurred” and for Year 1 the result was nearly the same. However, the findings from 
the School Visits Study provide some context to these TA Survey results. Findings from the 
school visits in Year 1 indicate that the severity of the technology-related problems were much 
less than those observed during the field test. Moreover, findings from the school visits in Year 
1 indicated that most technology-related problems were resolved within a couple of minutes or 
less. Consequently, even though the TA Survey results indicate that the prevalence of 
technology-related problems were similar in the field test and in Year 1, the additional 
contextual information obtained from the school visits indicate that the magnitude and 
duration of those technology-related problems were much less for Year 1. 
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TA Survey: Please indicate if any of the following technology-related problems occurred 
during testing. (Year 1 Results Compared to Field Test Results) 
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Figure 29. Percentage of TAs indicating particular types of technology-related problems occurred 
Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
*These questions did not appear on the field test version of the TA Survey. 
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As reported under Research Question 4, the percentage of students indicating that there were 
no technology-related problems during Year 1 was nearly 80% (for both the mathematics and 
ELA assessments); this was a substantive improvement from the field test results where only 
54% - 59% of students reported no technology-related problems on the EOY assessments in the 
field test (see Figure 30). This suggests that the actions taken by PARCC to help ameliorate the 
technology-related concerns discovered in the field test were beneficial. 


As with the TA Survey, the student test-takers were also asked to indicate which kinds of 
technology-related problems occurred while they were taking the test. Difficulty logging on was 
the most frequently reported problem with approximately one-fourth of students indicating 
that they encountered this problem on both the mathematics and ELA assessments (see 

Figure 30). This was also among the most frequently reported technology-related problems by 
TAs, although over 50% of TAs reported this as a problem. It is not surprising that smaller 
percentages of students reported technology-related problems and higher percentages of TAs 
reported technology-related problems given that students were responding to the survey based 
on their individual test experience whereas TAs were responding based on their experience of 
administering the assessments to all their students across all their test sessions. The 
percentages of students reporting that they encountered other technology-related problems 
were similar for the field test and Year 1. Even though these percentages were similar for the 
field test and Year 1, the findings from the School Visits Study indicate that the severity of these 
technology-related problems was much less in Year 1, which helps to explain why considerably 
more students reported “no technology-related problems” in Year 1. 
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Student Survey: Types of Technology-Related Problems Encountered 
(Year 1 Results Compared to Field Test Results for Math and ELA) 
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Figure 30. Percentage of students indicating particular kinds of technology-related problems occurred during test administration. 

It should be noted that the survey items were formatted differently on the Year 1 and the field test surveys. For the field test, the survey item was formatted as 
“Select all that apply” and for Year 1, the survey items were formatted as “Select One” (i.e., Did this problem occur? Y/N). 

*This response option was not included on the field test version of the Student Survey. This response option was added to the Year 1 Student Survey based on 
findings from the field test, which indicated that “difficulty logging on” was encountered during the field test. 
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Findings from the School Visits Study indicate that technology-related problems occurred in 29 
of the 109 observed test sessions (27%). The findings from the field test indicate that 
technology-related problems occurred in 68% of the observed test sessions; consequently, 
there was a substantial decrease in the occurrence of technology-related problems from the 
field test to Year 1, as evidenced from School Visits Study. Difficulty logging on was the most 
frequently observed technology-related problem in Year 1, which is consistent with findings 
from the TA Survey and Student Survey. Problems with the system disconnecting (“kicked out”) 
were also among the most frequently observed problems in Year 1; this is also consistent with 
findings from the TA Survey. The observers noted that the technology-related problems were 
resolved within minutes for nearly two-thirds of the sessions. In only two test sessions (1%) did 
the observers note that the technology-related problem took longer than 20 minutes to 
resolve. These findings are in contrast to the field test, where 17 of 61 observed schools (28%) 
experienced technical problems that took over 10 minutes to resolve. It took over 30 minutes 
to resolve technical issues in six of those schools and problems were so severe testing was 
suspended in four schools. 


The debriefing interview with TAs provided confirmatory information regarding the types of 
technology-related problems. Table 13 contains a breakdown of the types of technology- 
related problems TAs mentioned during the debriefing interviews. Again, the primary problems 
TAs identified were with students being kicked-off the system (10 interviews) and with difficulty 
logging students on or off the system (7 interviews). 


Table 13. Types of Technology-Related Problems Noted by TAs during Interviews 


Theme Sources? References?” 
Kicked out; reset quickly 10 11 
Login issues; resolved quickly 7 8 
JAVA issues; updated quickly 5 5 
Froze-up; restarted quickly 4 4 
Connectivity, more than 20 minutes to resolve 2 2 


comments made about that topic. 

In addition to the types of technology-related problems discussed above, the Student Surveys 
also asked students about some other features related to the technology of CBTs. Both versions 
of the Student Survey (mathematics and ELA) asked students the following questions about the 
tools: “Was it easy to use the highlighter tool?” and “Was it easy to make pictures or words 
bigger or smaller?” The results displayed in Figure 31 indicate that for the mathematics 
assessments approximately half (52%) of the students reported that they did not use the 
highlighter tool and 43% indicated that it was easy to use the highlighter tool (which was over 
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85% of those who used the tool); the results were similar for the field test, although for Year 1 
there was a slight trend of fewer “Did not use” ratings and more “Yes” ratings. For the ELA 
assessments fewer students indicated that they “Did not use” the highlighter tool than what 
was found for mathematics (41% “Did not use” for ELA compared to 52% for math). More than 
half of the ELA students indicated that “Yes” it was easy to use the highlighter tool (which was 
over 90% of those who used the tool), although that was down somewhat from the field test 
results (53% in Year 1 compared to 61% in the field test). 


Student Survey: "Was it easy to use the 


highlighter tool?" 
(Year 1 Results Compared to Field Test Results for Math and ELA) 
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Figure 31. Percentage of students indicating it was easy to use the highlighter. 

Percentages are based on the number of valid (non-missing) responses. 

Figure 32 displays the results for the Student Survey question, “Was it easy to make pictures or 
words bigger or smaller?” Most students (approximately 70%) indicated that they did not use 
this magnification tool for either the mathematics or ELA assessments. This was a somewhat 
higher percentage of “Did not use” ratings than what was found for the field test, particularly 
for ELA (62% of ELA students indicated they did not use the magnification tool on the field test). 
Of those who did use the magnification tool, a greater percentage reported that “Yes” it was 
easy to make pictures or words bigger or smaller than “No” (when those who did not use the 
tool were excluded, over 70% indicated that “yes” it was easy to use the magnification tool). 
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Student Survey: "Was it easy to make pictures or 


words bigger or smaller?" 
(Year 1 Results Compared to Field Test Results for Math and ELA) 


100.0% 
90.0% 
80.0% 
70.0% 
60.0% 
50.0% 
40.0% 
30.0% 
20.0% 
10.0% 

0.0% 


® Did not use 
mB No 


m Yes 


CBT EOY Math CBT EOY Math CBT EOY ELA CBT EOY ELA 
Year 1 Field Test Year 1 Field Test 
n = 3,563,623 n= 157,872 n = 3,503,474 n = 160,820 


Math 


Figure 32. Percentage of students indicating it was easy to make pictures or words bigger or smaller. 
Percentages are based on the number of valid (non-missing) responses. 


The CBT version of the math Student Survey asked students two additional questions about 
tools specific to the mathematics assessments—the calculator and the equation editor. 
Students were asked to indicate, “Was it easy to use the calculator?” and “Was it easy to enter 
math symbols and numbers for your answers?” The findings for those survey items are 
presented in Figures 32 and 33, respectively. The results in Figure 33 show that only about 10% 
indicated that it was not easy to use the calculator tool (with 43% indicating they did not use 
the calculator"); this result occurred for both Year 1 and the field test. The results in Figure 34 
show that 19% of students reported that it was not easy to enter symbols and numbers into the 
equation editor (with only 12% indicating they did not enter math symbols and numbers); this 
was a slight reduction from the field test (23% in the field test reported that it was not easy to 
enter math symbols and numbers). 


1 Student Survey results reflect responses from all grade levels. It should be noted that calculators are not used in 
Grades 3 —5, which likely accounts for the high percentage of “did not use” responses. When the “did not use” 
responses are excluded from the analysis, the percentage of students indicating that it was easy to use the 
calculator was over 80%. 
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Student Survey: "Was it easy to use the 


calculator?" 
(Year 1 Results Compared to Field Test Results) 
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Figure 33. Percentage of students indicating that the calculator tool was easy to use (Math). 
Percentages are based on the number of valid (non-missing) responses. Missing = 1,280. 


Student Survey: "Was it easy to enter math 


symbols and numbers for your answers?" 
(Year 1 Results Compared to Field Test Results) 
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Figure 34. Percentage of students indicating whether it was easy to enter math symbols and numbers 
(Math). 
Percentages are based on the number of valid (non-missing) responses. Missing = 1,280. 
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The CBT version of the ELA Student Survey asked students two additional questions specific to 
navigating within the ELA assessments. Students were asked to indicate, “Was it easy to move 
back and forth between passages or stories?” and “Was it easy to find information in the 
passages or stories when answering questions?” The findings for those survey items are 
presented in Figures 35 and 36, respectively. The results in Figure 35 show that the majority of 
students (79%) found it easy to move back and forth between passages/stories; the results 
from the field test were very similar. The results in Figure 36 also show that the majority of 
students (72%) found that it was easy to find information in the passages/stories, although a 
substantive percentage of students (23%) reported that “No” it was not easy to find 
information in the passages/stories; these results were similar to the results from the field test. 


Student Survey: "Was it easy to move back 


and forth between passages or stories?" 
(Year 1 Results Compared to Field Test Results) 
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Figure 35. Percentage of students indicating whether it was easy to move back and forth between 
passages or Stories (ELA). 
Percentages are based on the number of valid (non-missing) responses. 
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Student Survey: "Was it easy to find 
information in the passages or stories 


when answering questions?" 
(Year 1 Results Compared to Field Test Results) 
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Figure 36. Percentage of students indicating whether it was easy to find information in passages or 
stories (ELA). 

Percentages are based on the number of valid (non-missing) responses. 

In summary, the findings from the Student Surveys and the School Visits Study indicate that 
there was a substantial decline in the occurrence of technology-related problems from the field 
test to Year 1, thereby indicating greater adherence to Standard 6.4. While the TA Survey 
results indicate that similar percentages of TAs in the field test and in Year 1 reported 
technology-related problems, the contextual information obtained from the School Visits Study 
indicates that the severity of those technology-related problems was much less in Year 1, with 
most issues resolved within a few minutes or less. These findings suggest that the actions 
PARCC took since the field test to help improve the local technology experience (e.g., additional 
infrastructure trials prior to test administration, enhanced guidance and documentation for the 
SystemCheck tool, updated training modules on conducting infrastructure trials and proctor 
caching, a new technology preparedness training module, and enhanced communication to 
states and districts regarding technology preparedness) were beneficial (see PARCC Field Test: 
Lessons Learned, PARCC, 2015). The Year 1 findings also suggest that a more targeted focus on 
reconciling issues with getting students logged on and with the system disconnecting would be 
useful for the TAs. Finally, regarding the tools and features of the CBTs, the findings from the 
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Student Survey indicate that very few students have problems with the highlighter tool, the 
magnification tool, the calculator tool (mathematics) and navigating between passages/stories 
(ELA), which was consistent with findings from the field test. The survey findings do indicate 
that there are substantive numbers of students who are not using these tools/features, which 
suggests that more emphasis should be placed on encouraging students to take advantage of 
these tools and features. The majority of students also reported that it was easy to enter math 
symbols and numbers; however, nearly 20% reported difficulty using this feature. Similarly, the 
majority of students reported that it was easy to find information in the passages/stories when 
answering questions; however, 23% reported difficulty with this. These findings suggest that 
students might benefit from additional practice with these features. 


Findings for Research Question 7b: Are there any interruptions during the test administration 
related to students asking questions? What do students ask? 


All three of the studies provide information to help inform Research Question 7b. First, the TA 
Surveys (CBT and PBT versions) asked TAs to identify the types of questions students asked 
during test administration. This survey item addresses Research Question 7b while also 
addressing the claim that “Students respond to items as intended” (claim 4). Confusion about 
the test and interruptions caused by students asking TAs questions during test administration 
are both factors that could threaten the veracity of this claim. 


Figure 37 provides information on the prevalence of different types of questions students asked 
during CBT administrations. The most frequently asked questions, as indicated by TAs, were 
questions regarding technology-related problems and questions about exiting the test. Fifty- 
nine percent of TAs reported that students asked questions about technology-related problems 
and approximately half reported that students asked about how to exit the test. Figure 38 
provides this same information for PBT administrations. Overall, the TAs administering PBTs 
reported fewer questions from students during test administration. The most frequently asked 
questions were questions regarding clarification on the instructions the TAs read to their 
students and questions about how to mark answers and enter responses; roughly one-third of 
TAs administering PBTs indicated that students asked these kinds of questions during test 
administration. 
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CBT TA Survey: Please indicate if students asked questions about the following topics. 


(Select all that apply) 
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Figure 37. Percentage of TAs reporting students asked questions about various topics (CBTs). 

Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. These response options were identified from the themes in the open-ended comments provided by TAs on the field test survey. 
Consequently, no comparisons with the field test results are presented in this figure. 
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PBT TA Survey: "Please indicate if student(s) asked questions about the following topics." 
(Select all that apply) 
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Figure 38. Percentage of TAs reporting students asked questions about various topics (PBTs). 


Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. These response options were identified from the themes in the open-ended comments provided by TAs on the field test survey. 
Consequently, no comparisons with the field test results are presented in this figure. 
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The observations conducted for the School Visits Study also provide an indication about 
interruptions during test administration related to students asking questions. In the observer 
checklist, there was one task that focused on how TAs handled student questions during test 
administration. Observers indicated that students asked questions in 81 of 109 sessions (74%) 
in 79 schools. Of the 81 sessions, TAs handled student questions appropriately in 75 sessions 
(93%), which suggest that potential disruptions caused by students asking questions were likely 
minimized. It should be noted that students asked questions in 72 of 91 CBT sessions (79%) 
while students asked questions in nine of 18 PBT sessions (50%). There were only six sessions of 
81 (7%) where TAs received ratings of “Partially Met” or “Did Not Meet” on the observer 
checklist. An observer noted in one PBT session that the TA refused to listen to the student’s 
question, stating to the student “I cannot answer any questions.” Later, when asked about this 
exchange during the debriefing interview, the TA told the observer that she was informed in 
her training that TAs are not allowed to answer any questions students have about the test or 
even speak to them outside of reading the script. This TA expressed concern about jeopardizing 
her teaching license stating in the interview, “I’m not going to lose my license over this.” In 
three other sessions (CBT), observers noted that students asked about tools or navigation and 
TAs provided some guidance in using tools and the navigational process in order to answer the 
test questions; however, they did not help the students answer the content of test question. In 
comparison to the field test, observers found that students asked questions in 59 of 77 sessions 
(77%) which is comparable to Year 1; however, TAs received ratings of “Partially Met” or Did 
Not Meet” in eight of the 59 sessions (14%), compared to 7% for Year 1. 


There was also one interview question in the debriefing protocol addressing questions students 
asked during the test. In 34 of the 88 interviews TAs indicated a range of questions were asked 
by students regarding technology issues that include: scrolling within a document, using the 
highlighter tool, drag and drop items, and not realizing the text boxes will expand as they type 
more (see Table 14). There were only three interviews with TAs administering PBT that had any 
student questions. There were only a total of nine other student questions reported by TAs; 
those included questions about the purpose of the test, definitions of words or phrases, and 
topics unrelated to the test (e.g., “can | use the bathroom?”). 
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Table 14. Student Questions from TA Interviews 


Theme Sources? References® Sample Responses 


e Another student thought that an example number 
line was supposed to be used 

e Students are confused about how to enter the 
math symbols 

34 36 e A few students asked for clarification about the 

grid, across testing there have been a lot of 
confusion when questions have a graphic on one 
page and the question is on the following page‘ 

e plotting the points on a graph with 3 coordinates 


Navigational or how to 
enter responses 


Questions related to 4 5 e What does statistic mean? 

content e Student asked what natural logs are 
Questions not related 4 4 e Using bathroom or getting a break 
content e Why are we taking this test? 


indicates number of interviews providing information about that topic. "Indicates number of notes and/or 
comments made about that topic. ‘Response associated with the PBT. 

In summary, findings from the TA Surveys and the School Visits Study indicate that students 
commonly asked TAs questions during test administration, most notably for the CBT 
administrations. Findings from the School Visits Study indicate that the amount of questions 
students asked were similar for the field test and Year 1. From the TA Survey (CBT version) for 
Year 1, we know that more than half of the TAs reported that students asked questions about 
technology-related problems and questions about how to exit the test. Findings from the 
observations of test administration indicate that, when students asked questions, the TAs 
handled those questions appropriately in nearly all instances, which suggests that interruptions 
caused by students asking questions were likely minimized. To help further reduce 
interruptions caused by students asking questions, these findings suggest that additional clarity 
for exiting the text (perhaps in the TAs’ script and in the instructions presented on students’ 
monitors) should be implemented. Furthermore, the suggestions presented under the results 
for Research Question 7a to enhance training and guidance for minimizing and handling 
problems associated with students logging on and exiting the test should also help to reduce 
student questions about technology-related topics. These actions should help to further 
strengthen adherence to Standards 4.16, 6.4 and 6.5. 


Findings for Research Question 8: If any disruptions, interruptions, or other problems 
occurred, did the TAs deal with the issue appropriately and effectively? 


Research Question 7 focused on interruptions during testing (which is relevant to the claim that 
students respond to items as intended). Research Question 8 focuses on how 
effectively/appropriately TAs handled those interruptions. Thus, Research Question 8 is most 
relevant to the claim that, “TAs are prepared to administer assessments as intended” (claim 1). 
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For this claim to be true, TAs must have the knowledge and skills to handle interruptions, 
whether those knowledge and skills come from background experience and/or from training 
(Research Question 8a). 


As discussed in the results for several of the previous research questions, problems related to 
technology were the most common source of interruptions. Findings from the School Visits 
Study indicate that when technological difficulties occurred the TAs made appropriate attempts 
to resolve the issue by referring to their administration manuals, working with Technology 
Coordinators, and/or contacting the Help Desk. In no instance did the observers note that the 
TAs dealt with the technology problems inappropriately. Findings from the School Visits Study 
also indicate that when disruptive student behavior occurred, in the rare instances when it did 
occur, the TAs handled those instances appropriately. Finally, findings from the School Visits 
Study also indicate that when student questions arose, the TAs typically handled those 
questions appropriately. The results presented below under Research Question 8a shed light on 
whether TAs’ preparedness for handling these types of occurrences was informed by 
background experience and/or training. 


Findings for Research Question 8a: Were the TAs’ actions informed by background 
experience, by training, or a combination of the two? 


Regarding the potential for background experience to inform TAs’ actions for handling 
disruptions/problems, the TA Survey (CBT version) asked the TAs to indicate whether they had 
ever administered a computer-based test to students (either at the state or district level). The 
results in Figure 39 indicate that roughly half of the TAs who responded to the CBT TA Survey 
had previously administered a computer-based test to students at the district or state level. 
Consequently, the potential for background experience (related to administering CBTs) to 
inform TAs’ handling of disruptions and problems was relevant for roughly half of the TAs. This 
finding was comparable to the finding from the field test. 
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TA Survey: Have you ever administered a computer-based 
test to students before (either at the state or district level)? 
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Figure 39. Percentage of TAs indicating if they previously administered a computer-based test. 
Percentages based on valid (non-missing) responses. 

With regard to the potential for training to inform TAs’ actions for handling 
disruptions/problems, many of the results presented earlier under Research Question 2 are 
relevant for this research question as well. On average, approximately two-thirds of TAs 
administering CBTs reported that they found the PARCC on-line training modules to be effective 
in preparing them to administer the PARCC CBTs (see Figure 5 under Research Question 2). 
Similar results were found for TAs administering PBTs with approximately two-thirds of TAs 
reporting that the training modules on PearsonAccess"®t, Student Registration Import, and 
Student Readiness Resources effectively prepared them to administer the PARCC PBTs; even 
higher percentages of TAs administering PBTs (75% or more) indicated that the following PARCC 
training modules effectively prepared them to administer the PARCC PBTs: Introduction to 
PARCC, Accessibility Features & Accommodations, and Administration of Paper-Based 
Assessments (see Figure 7 under Research Question 2). Moreover, more than 80% of TAs 
administering CBTs reported that the PARCC tutorial helped them to better understand the 
tools and functionalities of the TestNav system (see Figure 10 under Research Question 2), and 
approximately 75% of TAs administering PBTs reported that the PARCC Paper-based Student 
Tutorials helped prepare them to administer the PARCC PBTs (See Figure 11). In addition, more 
than 80% of TAs (regardless of whether they administered CBTs or PBTs) indicated that the 
training they received from their school effectively prepared them to administer PARCC 
assessments (see Figure 8 under Research Question 2). Collectively, these results suggest that 
the TAs’ actions during test administration were likely to have been informed by the training 
they received. 


One area in which training may benefit from enhancements is with regard to training TAs to 
handle technology-related problems. Even though the majority of TAs (56%) indicated that the 
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PARCC online trainings prepared them to resolve basic problems related to technology (see 
Figure 6 under Research Question 2), this was a comparatively low percentage of respondents 
endorsing this survey item as compared to other survey items regarding the effectiveness of 
training. Moreover, only about 12% of the TAs reported that “no technology-related problems 
occurred” during their test administration experience, which underscores the importance of 
enhancing the training TAs receive on handling technology-related problems. As with the 
findings from the field test, the technology-related problems tended to be of a local nature 
rather than a system-wide nature. This indicates that training on technology-related issues 
should be sure to cover issues such as logging students in, dealing with firewall settings, 
adjusting computer settings, and so forth. 


In summary, findings from the studies indicate that TAs’ actions were informed by background 
experience and by training. For the most part, the TAs indicated that the various trainings they 
received were effective; however, there is evidence of a need for additional enhancements to 
training on handling local technology-related issues such as, logging students on and handling 
system disconnects so as to further minimize disruptions during testing (and thereby further 
strengthen adherence to Standards 6.4 and 6.5). It is important to note that PARCC has added a 
section to the manuals to provide additional emphasis and guidance on the importance of 
conducting an infrastructure trial to reduce the frequency of technology related problems. 


Findings for Research Question 8b: Would a different approach to handling the disruptions or 
problems that occurred during test administration been more effective? 


The results reported under Research Question 7b indicate that students often had questions 
(more often for CBT than PBT) during test administration; as reported under Research 
Question 2, results from the observation study indicate that in most instances the TAs handled 
the students’ questions appropriately and effectively. There were a few occasions, however, 
where observers noted that the TAs provided too much assistance on navigating through the 
assessment and one instance where the TA indicated that TAs were not to talk to students 
during testing (including responding to student questions). Also, when asked about suggestions 
for improving the TA Manual (see results for Research Question 1) one suggestion was to 
provide additional guidance in the TA Manual on how to respond to student questions. These 
findings suggest that TAs could be more effective in handling disruptions caused by students 
asking questions if they had additional guidance on how to respond to student questions. 


Finally, with regard to handling technology-related problems, during the School Visits Study 
observers found that several schools reduced the amount of staff resources needed to support 
CBT administration by conducting multiple CBT administrations in classrooms located in a 
common wing of the building. With this arrangement they were able to set-up and monitor 
testing through one administration computer located in the common hallway; the test 
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administration computer was staffed by school and/or district technology specialists. When TAs 
experienced technology-related problems, such as log on or connectivity issues, they could go 
to their classroom door and support was readily available. This is an alternative approach that 
some schools devised for handling technology-related problems, which appeared to be very 
effective while also minimizing staff resources. 


Findings for Research Question 9: Was security of test materials maintained at all times? 


There is considerable overlap between Research Question 9 and Research Question 6, “Are 
there any apparent attempts to record or copy test materials including the test questions by 
students or others?” Findings that are pertinent to test security were discussed within the 
results section for Research Question 6. Consequently, relevant findings for Research 
Question 9 are reported in the results section for Research Question 6. 


Findings for Research Question 10: Did the test administration create minimal disruption to 
the school and staff? 


For the testing system to be effective, it must not be overly burdensome on the schools’ 
resources. If test administration becomes overly burdensome, then it might not be feasible for 
schools to have sufficient resources and supports to administer the assessments (claim 2). 
During the debriefing interviews, the TAs were specifically asked if they felt administering 
PARCC required more time and resources than previous state tests. In over two-thirds of the 
interviews (59 of 88), the TAs responded, “yes.” TA comments typically indicated that the 
number of staff hours required for preparation was greater than what was required for the 
prior state tests. TAs also commonly stated that the amount of time computer labs were 
available for students’ other schoolwork decreased, especially in light of the PBA testing and 
the EOY testing window. In many schools, there were not enough computers/tablets for all 
grades to test at the same time (e.g., during the same week). Consequently, the PBA and EOY 
testing windows had to be stretched out over longer periods of time to allow all grades to 
complete testing on computers/devices. The two testing windows combined with having to 
extend testing over longer periods of time to ensure all students had a computer/tablet 
resulted in the “PARCC schedule being more disruptive to the curriculum schedule.” Also, based 
on observer notes and responses during interviews, it was evident that at least one additional 
person was needed to support the TA during testing, particularly for CBT administrations. 
Although observers did not specifically track the number of school staff involved in each 
observed session, checklist entries in over half of the observations indicated that more than 
one person was involved in the test session. These findings suggest that test administration 


|.” 


created disruptions that were more than “minimal.” However, when compared to the TA Study 
results from the field test the results from the Year 1 study indicate that many of the concerns 


and issues from the field test were successfully addressed by the actions PARCC took after the 
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field test (see PARCC Field Test: Lessons Learned, PARCC, 2015). Consequently, the burden 
placed on schools’ resources appears to be somewhat less in Year 1 than in the field test. 


Summary and Recommendations 


The overall summary and recommendations, based on the findings for the research questions, 
are discussed in this section by each claim in the TOA under the Administration phase. 


Claim 1: TAs are Prepared to Administer the Assessments as Intended 


In order for this claim to be true important assumptions that must be verified are: (a) TAs are 
trained to administer the assessments and (b) training is effective. Findings related to several of 
the research questions provide evidence for these assumptions and for the overall claim of 
preparedness. 


The research questions that most directly provide evidence for this claim and its assumptions 
are: 


e Research Question 2: To what degree do the TAs follow the protocols and instructions? 
(addresses Standards 6.1 and 6.2) 

e Research Question 6: Are there any apparent attempts to record or copy test materials 
including the test questions by students or others? (addresses Standards 6.6 and 6.7) 

e Research Question 8: If any disruptions, interruptions, or other problems occurred, did 
the test administrators deal with the issue appropriately and effectively? (addresses 
Standards 6.1 and 6.3) 

o Research Question 8a: Were the TAs’ actions informed by training, by background 
experience, or a combination of the two? (addresses Standard 6.1) 
o Research Question 8b: Would a different approach have been more effective? 

e Research Question 9: Was security of test materials maintained at all times? (addresses 

Standards 6.6 and 6.7) 


Test administrators’ preparedness to administer assessments as intended is directly related to 
the training they received. The investigations of the research questions and their underlying 
assumptions indicate that that the majority of TAs administering CBTs and PBTs reported that 
the PARCC online trainings, particularly the Introduction to PARCC module, effectively prepared 
them to administer the PARCC assessments. Even higher percentages of TAs reported that the 
training they received from their school effectively prepared them to administer PARCC 
assessments. In addition, most TAs reported that they gained additional familiarity with PARCC 
content via sample items, practice tests, and the PARCC tutorial on TestNav (CBT only) prior to 
test administration. Findings from the School Visits Study indicate that TAs followed 
standardized procedures during test administration with very few exceptions (i.e., using the 
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timing box), and that test security was maintained at all times with no apparent attempts to 
record or copy test materials. The close adherence to the standardized procedures also speaks 
to the effectiveness of the training and training materials provided to TAs. A notable 
improvement from the field test is that in only one of the 109 observed test sessions did the TA 
deviate from the test administration script. This is a substantial improvement from the field test 
study in which observers noted that in almost half of the observed sessions TAs did not follow 
the script in the TA Manual verbatim. This notable improvement is likely attributable to the 
improvements made to the PARCC trainings and to the TA Manual since the field test. 


With regard to the effectiveness of the training and training material on administering AFAs, 
the vast majority of TAs reported reading the AFA Manual prior to test administration; this was 
a notable increase from the field test. Moreover, the majority of TAs agreed that the AFA 
trainings and materials effectively prepared them to administer AFAs. Observer ratings from 
the observed accommodation sessions also indicate that in nearly all of the observed 
accommodation sessions the TAs performed the tasks as intended. 


There were some areas, related to training and training materials, where the study findings 
indicate that additional improvements may be beneficial. First, just over half of the TAs 
indicated that the PARCC trainings/materials effectively prepared them to handle basic 
technology-related problems (CBT). While still a majority, this result compares less favorably 
than other findings regarding the effectiveness of trainings/materials (PARCC has made the 
following changes to all manuals for the 2015 Fall Block administration with additional 
improvements planned for the Spring 2016 administration: the TC manual was reduced by 46 
pages, the TA manual was reduce by more than half the pages, and restructured the manuals to 
tasks before, during, and after testing to better align with local practices). Second, only about 
one-third of TAs administering PBTs indicated that they worked with the PARCC paper-based 
student tutorials prior to test administration. Third, most TAs were unaware of whether the 
AFAs that were pre-identified in the PNP were made available to students during a practice 
session, and most were also unaware as to whether changes were made to a student’s PNP or 
AFAs during test administration. Just over a third of TAs reported that AFAs that were pre- 
identified in the PNP were made available to students during a practice session. Finally, while 
TAs were generally observed handling student questions appropriately and effectively, there 
was some evidence to indicate that additional training and guidance on how to handle student 
questions would be beneficial. 


In summary, the majority of the evidence from the research studies provides support for 

Claim 1—TAs are prepared to administer assessments as intended. Notable improvements were 
evidence from the field test to Year 1, particularly with the percentage of TAs following the 
scripts in their TA Manual. This improvement is likely attributable to the actions PARCC took 
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since the field test to improve the clarity of the TA Manual and to improve the effectiveness of 
the TA training, including the addition of more opportunity to interact with PARCC content prior 
to test administration. 


Even though the majority of evidence indicates support for Claim 1, there are findings to 
suggest that the support for Claim 1 could be further strengthened by taking into consideration 
the following recommendations: 


e Ensure that schools are well-informed in all aspects of PARCC test administration well in 
advance of the testing windows. The type of training that received the highest 
effectiveness rating was the training provided at the school-level. This underscores the 
importance of ensuring that schools have all the resources and materials they need to 
effectively train their staff well in advance of the testing window. This includes ensuring 
that the finalized TA Manuals are available in a timely manner. 

e More emphasis should be placed on ensuring that all TAs complete the PARCC tutorials 
prior to test administration, particularly for TAs administering PBTs. Of those who 
completed the tutorials, the vast majority agreed that the tutorials helped better 
prepare them to administer the PARCC assessments. Nonetheless, approximately one- 
third of TAs administering CBTs indicated that they did not review the PARCC tutorials 
prior to test administration and two-thirds of TAs administering PBTs indicated that they 
did not review the PARCC tutorials prior to test administration (it should be noted that 
PARCC has placed additional emphasis on the tutorials and practice test sections in the 
manual to further emphasize the importance of administering these to students prior to 
testing. PARCC has also created a pre-administration guidance document that has a 
whole section devoted to instructional supports and to encourage the use of tutorials 
and practice tests). 

e Enhance the training on handling basic technology-related problems. Only slightly more 
than half of the TAs agreed that the PARCC online trainings effectively prepared them to 
resolve basic problems related to technology during test administration. Particular 
attention should be given to handling local technology-related issues with logging 
students onto the assessment and with students getting kicked off the assessment as 
these were the two most frequently reported types of technology-related problems 
encountered during test administration. One area in which PARCC may wish to focus is 
on increasing the number of TAs who participate in the technology-oriented training 
modules. Between 35% - 40% of the TAs reported that the following modules were “not 
applicable” to them: Personal Needs Profile (PNP) Training, Proctor Caching & TestNav 
Configuration Training, Student Registration Import Training, and Technology 
“Readiness” for Schools and Districts Training. Perhaps these trainings are relevant to 
more TAs than those who actually completed them, particularly if any of these trainings 
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cover how to handle technology-related problems (it should be noted that PARCC has 
changed the PearsonAccess"*™t module to be broken into smaller, task-based modules. 
The aim of this change is that TAs will be able to view the shorter modules just prior to 
testing to refresh their memory on basic technology functions for testing. PARCC also 
added a new troubleshooting computer-based errors section to the manuals with direct 
links to common error messages to help provide transparency and a direct link to 
support information). 

e Provide additional guidance to TAs on how to handle student questions that arise during 
testing. In particular, provide guidance on how to handle student questions on 
navigating through the test. 

e Rethink the guidance provided in the TA Manual for the use of the timing box during 
test administration. Many TAs observed in the field test Study and in the Year 1 Study 
did not use the timing box exactly as it is described in the TA Manual. For example, 
many TAs provided verbal updates on time remaining as opposed to writing the time 
remaining in a timing box. Also, many TAs provided more frequent updates on time 
remaining than what is specified in the TA Manual. If flexibility in managing time is 
acceptable, consider revising the TA Manual to indicate that the timing box is one 
example of how to manage the time during the test session. Otherwise, if flexibility in 
managing time is not desirable, then consider providing more definitive instructions in 
the TA Manual on how time must be managed via use of a timing box (it should be noted 
that PARCC has added an improved materials/timing box to the script to add clarity for 
tracking time during testing). 

e Improve the process for identifying AFAs in advance via the Personal Needs Profile (PNP) 
so that it is clearer and easier to follow. Also, the majority of TAs responded either “no” 
or “not sure” when asked whether all the AFAs identified in the PNP were made 
available to students during a practice session. Consequently, PARCC may want to 
incorporate some additional scrutiny to ensure that students who need AFAs are indeed 
given the opportunity to practice with those AFAs prior to actual test administration (/t 
should be noted that PARCC has changed the PNP process to be identified during the 
student registration process so that this information is gathered earlier. Additionally, the 
PARCC Accessibility Features and Accommodations Manual has been revised to include a 
whole section devoted to the tasks to be completed before, during, and after testing for 
each accommodation. This revision more clearly outlines the tasks that need to be 
completed related to accommodations. PARCC has also placed additional emphasis on 
the tutorials and practice test sections in the manual to further emphasize the 
importance of administering these to students prior to testing. PARCC also created a pre- 
administration guidance document that has a whole section devoted to instructional 
supports and to encourage the use of tutorials and practice tests.) 
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Claim 2: TAs Have the Resources and Supports to Administer the Assessments 


For this claim to be true important assumptions that must be met are: (a) resources and 
supports must be clear, sufficiently detailed, and easy to follow, and (b) resources and supports 
must not be overly burdensome for TAs to use and apply. 


The research questions that most directly provide evidence for this claim and its assumptions 
are: 


e Research Question 1: To what degree do the test administrators (TAs) find the 
instructions clear, sufficiently detailed, and easy to follow? (addresses Standard 4.15) 

e Research Question 10: Did the test administration create minimal disruption to the 
school and staff? 


The instructions in the TA Manual serve as the TAs’ primary resource for administering the 
assessments. The vast majority of TAs for Year 1 (over 80%) agreed that the policies and 
procedures in the TA Manual were easy for them to understand. This is an increase from the 
field test where 71% of CBT administrators and 67% of PBT administrators agreed that the 
policies and procedures in the TA Manual were easy for them to understand. Similarly, the vast 
majority of TAs for Year 1 (85% or more) agreed that the instructions (including the scripts) in 
the TA Manual were easy for them to implement, whereas findings from the field test indicated 
that less than two-thirds of TAs (55% - 63% for PBT and CBT, respectively) agreed that the 
instructions in the TA Manual were easy for them to implement. Qualitative feedback obtained 
from TAs in Year 1 also indicated notable improvements since the field test in the usefulness of 
the TA Manual and, in particular, the usefulness of the scripts. There were still some 
suggestions from TAs to further streamline the TA Manual and the scripts and suggestions to 
further clarify the instructions for navigating between sessions and for ending sessions and 
exiting the test; however, these suggestions were much less prevalent in Year 1 than in the field 
test. 


For the testing system to be effective, it must not be overly burdensome on the schools’ 
resources. If test administration becomes overly burdensome, then it might not be feasible for 
schools to have sufficient resources and supports to administer the assessments. While there 
were fewer requests to streamline the materials (notably the TA Manual) in Year 1 than in the 
field test, many (over two-thirds) stated that the administration of the PARCC assessment 
required more time and resources to administer than previous state tests. The TAs and TCs 
most commonly indicated that the number of staff hours required for preparation was greater 
for the administration of the PARCC assessment. The TAs and TCs also commonly stated that 
PARCC was more disruptive to the curriculum schedule than prior state tests. The TAs and TCs 
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explained that this was due having two testing windows (one for PBA and one for EOY) and that 
each testing window was stretched out over a longer period of time because there were not 
enough devices for all students to test at the same time. 


In summary, the evidence collected from the studies indicates that the TA Manual, which is the 
primary resource for administering the PARCC assessments, is clear, sufficiently detailed, and 
easy to implement. These results suggest that the actions implemented by PARCC since field 
test to improve the TA Manual such as, shortening the TA Manual by reducing redundancies, 
adding grade band scripts, adding checklists of tasks, adding additional graphics and icons to 
increase user understanding, clarifying terminology, etc. (see PARCC Field Test: Lessons 
Learned, PARCC, 2015) were beneficial and favorably received by the TAs. This evidence helps 
support the claim that TAs have the resources and supports to administer PARCC. Other 
evidence, however, indicates that test administration often creates more than minimal 
disruption to the school and staff, most notably with regard to the amount staff hours required 
to prepare for test administration and with regard to the amount of time required for testing. 
Consequently, the collective set of evidence indicates mixed support for Claim 2. 


The primary recommendation to help further strengthen support for Claim 2 is one that PARCC 
has already implemented, which is to reduce the amount of time required for testing. This is 
being accomplished by eliminating the two testing windows, one for PBA and one for EOY. 
Moving to a single testing window should help reduce the staff resources and time resources 
required for the PARCC assessment, and thereby help to minimize the disruption caused by test 
administration. 


Claim 3: Technology Improves and Facilitates the Assessment Experience 


For this claim to be true an important assumption that must be met is that the technology must 
work as intended. If the technology does not work as intended—for example, if there are 
functionality problems and/or delays in its application, then the veracity of this claim is 
threatened. 


The evidence for this claim comes directly from Research Question 7a: 


e Are there any technology-related problems during test administration? (addresses 
Standards 6.3 and 6.4) 


Findings from the TA Survey indicate that most TAs reported that they encountered one or 
more problems related to technology during the test administration. These findings were 
similar to the findings from the field test. However, the findings from the School Visits Study 
provide additional context. Findings from the school visits in Year 1 indicate that fewer 
problems with technology were observed during test administration and that when problems 
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did occur the severity of the problems was much less than those observed during the field test. 
Observers noted that most problems were resolved within a couple of minutes. Consequently, 
even though the TA Survey results indicate that the prevalence of technology-related problems 
were similar in the field test and in Year 1, the additional contextual information obtained from 
the school visits indicate that the magnitude and duration of those technology-related 
problems were much less for Year 1. 


Students reported encountering fewer problems with the technology than TAs. Students also 
reported substantially fewer problems with the technology in Year 1 than in the field test. 
Reports of technology-related problems might have been lower for students than for TAs 
because students were asked about problems that occurred during their own individual test, 
whereas TAs were asked about problems across all students in their administration session(s). 
Another reason that students may have reported fewer technology-related problems than TAs 
is that the students may have been less aware of the problems given that the problems were 
typically minor and resolved within a couple minutes. 


The TAs reported that they most frequently encountered problems with logging into the 
assessment and with the system disconnecting/logging off. Students also reported “difficulty 
logging on” as one of the most frequently encountered technology-related problems. Similar 
percentages of students also reported difficulties with dragging or moving objects on the 
screen. With regard to the tools and features of the CBTs, very few students reported having 
problems with the highlighter tool, the magnification tool, the calculator tool (mathematics) 
and navigating between passages/stories (ELA), which was consistent with findings from the 
field test. The findings do indicate that there are substantive numbers of students who are not 
using these tools/features. The majority of students also reported that it was easy to enter 
math symbols and numbers, although nearly a fifth of the students reported difficulty using this 
feature. Similarly, the majority of students reported that it was easy to find information in the 
passages/stories when answering questions on the ELA assessment, although nearly a fourth 
reported difficulty with this. 


Overall, the findings suggest that the actions PARCC took since the field test to help improve 
the local technology experience (e.g., additional infrastructure trials prior to test 
administration, enhanced guidance and documentation for the SystemCheck tool, updated 
training modules on conducting infrastructure trials and proctor caching, a new technology 
preparedness training module, and enhanced communication to states and districts regarding 
technology preparedness) were beneficial (see PARCC Field Test: Lessons Learned, PARCC, 
2015), and that there is stronger support for Claim 3 in Year 1 than in the field test. 


Recommendations for further strengthening support for Claim 3 include: 
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e Provide a more targeted focus on reconciling local technology-related issues with 
getting students logged on and with the system disconnecting. (/t is important to note 
that PARCC has added a section to the manuals to provide additional emphasis and 
guidance on the importance of conducting an infrastructure trial to reduce the frequency 
of technology related problems.) 

e Place more emphasis on ensuring that students take advantage of the tools embedded 
within the CBTs given that substantive numbers of students indicated that they did not 
use these tools and features. 

e Provide students with additional practice on entering math symbols and numbers prior 
to test administration. (/t should be noted that PARCC has developed a standalone 
equation editor for teachers to use with students for advanced practice.) 

e Provide students with additional practice on finding information in passages/stories on 
the ELA assessment prior to test administration. 


Claim 4: Students Respond to Items as Intended 


Finally, for this claim to be true, several assumptions must be verified. First, for students to 
respond to items as intended, they must understand the directions read by the test 
administrator (i.e., the script). Second, students must also understand the directions they read 
for the questions on the test. If the students do not understand the directions read by the TA 
and/or if they are confused by the directions they read on the test, then they may not respond 
to the items as intended. Third, to respond to items as intended, students need to have some 
engagement with the assessment. If they are not engaged or if they have a low level of 
engagement, then they might not sufficiently attend to items so as to respond to them as 
intended. Many factors can negatively impact student engagement, such as (a) too little time to 
complete the assessment, (b) lack of familiarity with the content being assessed, (c) lack of 
familiarity with the mode of assessment, (d) distractions caused by unauthorized visitors, (e) 
distractions caused disruptive student behavior, and (f) technological problems. These are some 
of the factors investigated in this study. 


The research questions that provide evidence for Claim 4 and its assumptions are: 


e Research Question 3: Do the students appear to understand the instructions provided to 
them? (addresses Standards 4.16 and 6.5) 

e Research Question 4: To what degree are students engaged in taking the test? 
(addresses Standards 4.16, 6.3, 6.4, and 6.5) 

e Research Question 5: Is there any disruptive student behavior during the session? 
(addresses Standard 6.4) 

e Research Question 7: Are there any interruptions during the session? (addresses 
Standards 6.3 and 6.4) 
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o Research Question 7a: Are there any technology-related problems during test 
administration? 

o Research Question 7b: What type of questions, if any, do students ask during the 
test administration? (also relevant to Standards 4.16 and 6.5) 


Findings from the Student Surveys show that the vast majority of students report that they 
understood the directions (i.e., script) read by their TA. This finding was similar to the finding 
from the field test. Converging evidence was found from the TA Survey results. The vast 
majority of TAs agreed that students appeared to understand the instructions they read to 
them and that the instructions covered all of the information necessary to take the test. These 
TA Survey results reflect a notable increase from the field test to Year 1, which suggests that 
the improvements that PARCC made to the script in the TA Manual since the field test were 
beneficial. This evidence helps to verify the assumption that students understand the directions 
read by the TA. 


With regard to students’ understanding of the directions for the questions on the test, the 
Student Survey results indicate that the majority of students (approximately half) reported that 
it was hard to understand the directions for the questions on the test “some of the time” and 
nearly a third of students taking the mathematics assessments and nearly a fourth of students 
taking the ELA assessments indicated that it was hard to understand the directions for the 
questions on the test “most or almost all of the time.” These findings were similar for the field 
test. This evidence indicates a lack of support for the assumption that students understand the 
directions for the questions on the test. Per Standards 4.16 and 6.5, students are more likely to 
understand test instructions if they have had practice with the items and with the mode of 
testing (e.g., computer-administered) prior to actual testing. Most students reported practicing 
on a computer/tablet one or more times prior to actual testing, which was an increase from the 
field test. The TAs administering CBTs indicated that almost two-thirds of the students practiced 
with sample items and a practice test prior to actual testing. However, only about half of the 
students completed the tutorial prior to taking the computer-based assessment (as reported by 
TAs). With regard to the PBTs, the TA Survey results indicate that very few students completed 
the tutorial directed towards paper-based assessments. 


In terms of student engagement, the findings from the research studies provide information on 
several potential factors that might negatively impact student engagement. Findings reveal that 
the TAs effectively kept distractions such as, outside visitors and disruptive student behaviors 
to a minimum. The findings also indicate that technology-related problems were not likely to be 
a major disruption, as they were in the field test, given that most issues were minor in nature 
and resolved within a couple minutes. The majority of TAs administering CBTs reported that 
students asked questions about problems with technology and about how to exit the test, 
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although the findings from the School Visit Study indicate that in nearly all instances the TAs 
handled the student questions appropriately, thereby suggesting that distractions caused by 
students asking questions were minimized. 


Other factors that might negatively impact student engagement include familiarity with the test 
content, familiarity with the mode of assessment, and difficulty of the test. The findings 
indicate that most of the questions on the test covered topics students had already learned 
about in school, and most students were very familiar with the mode of assessment (CBT) and 
they preferred that mode of assessment. Moreover, feedback obtained from the TAs in the 
debriefing interviews and comments made by observers on the observation checklist indicated 
moderate to high levels of student engagement. 


In summary, the majority of the evidence indicates reasonable support for Claim 4—Students 
respond to items as intended. There were only two factors that were investigated for which the 
evidence threatens the veracity of the claim that students respond to items as intended. First, 
most students reported that it was hard to understand the directions for the questions on the 
test; if students do not understand the directions, then they might not be responding to items 
as intended. Second, test difficulty emerged as a potential factor that might have negatively 
impacted student engagement, particularly for the mathematics assessments. Most students 
reported that the mathematics assessments were harder than their school work. 


Recommendations for further strengthening support for Claim 4 include: 


e Improve the directions on the test, such as simplifying the instructions for answering the 
questions, to enhance students’ understanding. Findings indicated that many students 
asked questions about where to mark and enter their responses and how to find and 
use tools, which suggests that these might be aspects of instructions that may benefit 
from additional clarification. If students are better able to understand the instructions 
on the test, then the test might not seem so difficult to students. 

e Encourage greater participation in opportunities to practice with PARCC content prior to 
test administration, particularly completion of the student tutorials, as results indicate 
that fewer students practiced with tutorials than with sample items and practice tests. 
Greater familiarity to the PARCC content prior to test administration may also help 
students better understand the instructions on the test. 


Finally, it is worth noting that that the primary factors that emerged in the field test as 
impediments to student engagement— overly lengthy and redundant test instructions read by 
the TAs and delays caused by technology-related problems—appeared to have been 
ameliorated since the field test, as these did not emerge as significant impediments to student 
engagement in Year 1. 
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Conclusion 


The findings presented in this report do not provide an exhaustive investigation of the validity 
evidence for the PARCC test administration. Consequently, the evidence presented here should 
not be taken as the final word on the PARCC test administration. Nonetheless, the findings 
presented in this report do represent the current validity argument for the PARCC test 
administration. Overall, the findings from the Test Administration Study for Year 1 represent 
notable improvements in the validity evidence collected from the field test. In particular, the 
findings indicate stronger support for all four of the test administration claims in the Theory of 
Action: 


e Claim 1: TAs are prepared to administer the assessments as intended. 

e Claim 2: TAs have the resources and supports to administer the assessments. 
e Claim 3: Technology improves and facilitates the assessment experience. 

e Claim 4: Students respond to items as intended. 


The primary threats to the veracity of these claims, as identified through this Test 
Administration Study, are: 


e Continued problems of a local nature with logging students on and with students getting 
kicked-off the system. 

e The amount of time and staff required for administering PARCC assessments has been 
overly burdensome on schools’ resources. 

e Many students are having difficulty understanding the directions on the test for 
answering test questions. 


These concerns should be addressed for future operational assessments, as threats to the 
veracity of the test administration claims may undermine the validity of subsequent goals in the 
PARCC theory of action (i.e., valid scoring, reporting and use of test scores). Several 
recommendations for addressing these threats have been offered for consideration. 


Finally, it should be noted that a memo of preliminary findings from this Test Administration 
Study was submitted to PARCC leadership in June 2015. As a result of the findings and 
suggestions provided in this preliminary memo, PARCC has already made changes to the 
manuals that will be used for the 2™ operational year. Those improvements made by PARCC 
include: 
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. The manuals were streamlined, reducing a great deal of redundancy. 

. The Fall/Winter Test Coordinator Manual was reduced by 46 pages compared to last 
year. 

. The Test Administrator Manuals were reduced by more than half the pages compared to 
the 2015 Spring Administration (now 59-64 pages). 

. The structure of the manuals was changed to better align to local practices. These 
changes include restructuring the sections in the TCM and TAM to tasks before, during, 
and after testing and separating the CBT and PBT sections in the TCM. 

Additional content was added to further clarify procedures around using 
PearsonAccess"™t and after testing tasks. 

Policies were analyzed and adjusted accordingly to feedback from the field, including 
increasing the testing multiple classrooms flexibility and no longer requiring students to 
write names on scratch paper. 

PARCC is already working on the spring manuals, which will include great reductions to 
the student directions and the administrator scripts. 
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Appendix A — Observation Checklist for Test Administration Site Visits 
CBT Administrations 


PARCC Operational EOY Computer-Based Test (CBT) Observation Checklist 


Status Codes: (ez: 0900, ) 
1= Met 
2 Paria Met el 
oT Not Observed 
Step eee coanraadiae specific to line item 
ti 


Pre-Assessment Acti 


Prepare testing room by providing adequate space to work, seating every other 
seat, back to back, semicircle, or use partition/curtain/privacy screen) 


Prepare Test Administrator computer in location with students to be able to 
better manage and monitor test. Rate 2-partial if outside testing room. 


Prepare student computers for testing. (close browsers, programs, any 
programs that automatically launch (antivirus, updates), launch TestNav) 


Ensure prohibited materials are not visible. (posters, maps, number lines, word 
lists, formulas, definitions) Rate 1 if visible and not test related. 


TA confirms student roster matches correct Student Authorization Tickets and 
seal codes for this session. 


Draw timing box on the board, as in manual, but wait to fill it in until reading 


script (rate 2 if no wait). (see Timing box & Session times tab) 


Allow authorized visitors only, such as PARCC staff, state/local/school 
assessment monitors. (List title of non-school visitor, ask TA if needed) 


Observer Cues (for notes): Security breach - TA or student takes photo of computer screen (immediately text Richard). Also, if TA leaves room 


16 unattended while materials are not secure, coaches students or explains items, reads items/passages text Richard from car. 


Notes: 
17 
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Administer Assessment Activities (complete for both regular and accommodation sessions) 


Students arrive and leave belongings in central area or puts under seat. (No 
cell phones, other electronics, reference books, math formulas) 


After being seated at a computer, provide students with the following items: 


Scratch paper (blank, lined, or graph paper) and pencil. Students can request 
more. Students print name in upper right-hand corner of paper. 


Printed copy of the math reference sheets are allowed. (not required) \f not 
distributed rate 5-not applicable 


Hand-held calculators (TestNAv supplied or approved) (if none used, 5-NA): 
* Grades 3-5 - students with calculator accommodation only 

* Grades 6 -7 - Four function calculator for calculator portion of test only 

* Grade 8 - Scientific calculator for calculator portion of test only 

* Grade HS - Graphing calculator for calculator portion of test only 

Try best to see, if suspect wrong calc. type, Rate 3-NM & provide details 


Headphones: Required for PBA ELA test sessions (except grade 8)but not EOY. 
Students with text to speech accommodation may use headphones. 

NOTE: Student can use headphones as noise buffer, in which case the TA must ensure they are not plugged in. If TA does not verify, rate 
as 2-partial and provide a comment, if no one uses rate 5-NA. 


Aaminister Assessment ACctivities, CONt. 


Ensure students log in and have access to the test. (note how long it takes) 


Provide seal code and write Start/Stop times in timing box when script advises. 
(If only does verbal, rate 2-partial, and provide note) 


Read script verbatim in "SAY" box. (can't modify, but repeat and some 
clarification after reading verbatim first OK) (If a few slips, rate 1-Met) 
Handle technical problems as trained: (rate a and b separately, but if no issues rate 5-NA for both a and b) 


Individual: Note time to adjust remaining time, try to resolve, if not timely 
then move student to another computer or dismiss student to resume later 


Multiple students: pause testing, note time to adjust remaining time, try to 
resolve, TA must update timing box with new times 


Monitor student progress using administration computer. (If they ever look at 
it once testing begins, then rate 1-met) 


Monitor students by circulating around the room. 


42 16 |TC monitors test administration. (Rate 1-met, if in room at some point) 


Observer Cues: Are students engaged while TA reads script prior to starting test (too long, repetitive, clear, right length)? Students appear 
comfortable with navigation, tools, accessing and viewing graphics and videos? TA's can not help students verbally or to navigate (hand over 
43 hand). Document if testing is suspended due to technical problems .(e.g., length of time, actions taken) 
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Administer Assessment Activities, cont. 


Keep track of time accurately during the test session. (see Timing box & 
Session times tab) 

Answer student questions following script guidance. (note questions, if 
possible) 


Handle student alert to unansweable or misprinted test item. (TA notes 
content, course/grade, form #, item# for TC, does not write item text, does not 
allow other school staff to view item) (rate 3-NM if any part is missed, provide 
specific notes of actions) 


Provide a verbal prompt when 10 minutes remain. 


Provide a group 3 minute break if necessary (note if TA doesn't adjust test 
time on board) or one by one individual restroom breaks (computer screens 
must be turned off or apply visual block to monitor). Students must not 
communicate during breaks. Rate 3-NM if any part is missed, provide specific 
details. 


Handle testing irregularities. (fire drills, weather, threat) by reporting to TC, 
noting the time, if leave room lock door to testing room, document time 
starting back. (no interruptions, rate 5-N/A) 


Handle student misconduct appropriately, such as disruptions, cheating, 
accessing electronic device. (Dismisses student and exits their test) 


Administer Assessment Activities, cont. 


Allow students finishing early to submit test and leave, or stay and sit quietly 
or read a book, magazine, or textbook. (Arkansas requires students to remain 
in room until all students finish) 


Accommodations (Special Education or English Learner) 


Students receive accommodation in regular session. (Extended time should 
be in separate session) Rate 5-NA if none, Rate 2-partial if Ext Time 


Provide the number of students receiving accommodations in box to right. 


Students receive separate accommodation. (students receive same 
accommodations are grouped together) Rate 5 - NA if none 


List the accommodations being provided in box to right. 


Observer Cues: Note if anyone receives extended time and how it is administered (i.e., others dismissed or wait), may need to ask TA if any 
students have accommodations in regular session (e.g., text to speech, magnify), note if text to speech works well, or not well, and if there 


ga. Were any issues or difficulties with the human reader, particularly for math. 


62 
me 


Notes: Observers use accessibility and accommodation guide while observing separate session, note discrepancies. 


Post-Assessment Activities 


Collect Student Authorization Ticket and any test materials. (10a-e) If any item 


is missed, rate 3-NM and list item(s) missed in box to right. 


6g 31 [Return testing materials to the Test Coordinator. (we may not see) | | 


Intends (observer asks) on completing Test Administrator survey after he or 
she finishes all his or hers testing. Yes - rate 1, No or maybe - rate 3 


68 ‘Additional Comments/Notes: 


169 
70 
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PBT Administrations 


Status Codes: 

1= Met 

2= Partially Met 
3= Did Not Met 
4= Not Observed 
5 = Not Applicable 


| Status | Comments/Notes specific to line item 


Pre-Assessment Activities 


Prepare testing room by providing adequate space to work and sufficiently 
separated for secure testing. 


Ensure prohibited materials are not visible. (posters, maps, number lines, 
word lists, formulas, definitions) Rate 1 if visible and not test related. 


Draw a timing box on the board, as provided in manual, but wait to fill it in 
until reading script. (see examples on tab or last page of observation form) 


Post the example of the STOP and GO ON signs. a 


Allow authorized visitors only, such as PARCC staff, state/local/school 
assessment monitors. (List title of non-school visitor, ask TA if needed) 


Administer Assessment Activities (complete for both 7 Flalei-letete)iiliiveler-ltce)a) et 


Students arrive and leave belongings in central area or puts under seat. 
(No cell phones, other electronics, reference books, math formulas) 


7 |Verify student roster (attendance) after being seated. <> 


Observer Cues (for notes): Security breach - TA or student takes photo of test materials (immediately text Richard). Also, if TA leaves 
47 room unattended while materials are not secure, coaches students or explains items, reads items/passages text Richard from car. 


Administer Assessment Activities, cont. 


Read script verbatim in "SAY" box. (can't modify, but repeat and some 
clarification after reading verbatim first OK) (If a few slips, rate 1-Met) 


Distribute the following materials when indicated in the script: 


Test Bookie Se (a 


Scratch paper (blank, lined, or graph paper) and pencil. Students can 
request more. Students print name in upper right-hand corner of paper. 


Hand-held calculators (TestNAv supplied or approved) (if none used, 5-NA): 
* Grades 3-5 - students with calculator accommodation only 

© Grades 6 -7 - Four function calculator for calculator portion of test only 

* Grade 8 - Scientific calculator for calculator portion of test only 

* Grade HS - Graphing calculator for calculator portion of test only 

Try best to see, if suspect wrong calc. type, Rate 3-NM & provide details 
Math reference sheets (all but G3-4) & ruler/compass/protractor are 
rigs for G8 students and those taking Geometry and Alg, Integtd. Math 
Replace torn or damaged test booklet, but if detected during test, the 
student answers must later be transcribed in the replacement booklet. 
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Administer Assessment Activities, cont. 
Write Start/Stop times in timing box when script advises. (If only does 
verbal, rate 2-partial, and provide note) 


Handle contaminated (e.g., cut, illness) test materials appropriately by 
stopping test, medical attention, record time/item/page #, seal materials 
in plastic bag, clean area, if student returns replace booklet. 


Keep track of time accurately during the test session. (see tables on tab or 
last pages of checklist) 


Answer student questions following script guidance. (note questions, if 
possible) 


Handle student alert to unanswerable or misprinted test item. (TA notes 
content, course/grade, form #, item# for TC, does not write item text, does 
not allow other school staff to view item) Rate 3-NM if any part is missed 
and provide specific notes of actions. 


Observer Cues: Are students engaged while TA reads script prior to starting test (too long, repetitive, clear, right length)? Document if 
testing is suspended for student or all students (e.g., nature of incident, length of time, actions taken). Detail any breach in test security 
(e.g., TA discussing items, dismissing disruptive students, providing too much help to student). Note any evidence that students may be 


39 confused with different item types (possible indicator they did not access the practice tests or tutorials). 


Notes: Observers use accessibility and accommodation guide while observing separate session, note descrepancies. 


Administer Assessment Activities, cont. 


Provide a verbal prompt when 10 minutes remain. 


Provide a group 3 minute break (note if TA doesn't adjust test time on 
board) if necessary or one at a time individual restroom breaks (booklets 
must be closed or covered). Students must not communicate during 
breaks. Rate 3-NM if any part is missed, provide specific details. 

Handle testing irregularities. (fire drills, weather, threat) by reporting to 
TC, noting the time, if leave room lock door to testing room, document 
time starting back. (no interruptions, rate 5-N/A) 

Handle student misconduct appropriately, such as disruptions, cheating, 
accessing electronic device. (Dismiss student & collects test materials) 
Allow students finishing early to turn in test materials and leave, or stay 
and sit quietly or read a book, magazine, or textbook. (Arkansas requires 
students to remain in room until all students finish) 


Accommodations (Special Education or English Learner) 
Students receive accommodation in regular session. (Extended time 
should be in separate session) Rate 5-NA if none, Rate 2-partial if Ext Time 


Students receive separate accommdation. (students receive same 
accommdations are grouped together) Rate 5 - NA if none 
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Accommodation Activities, cont. 


Observer Cues: Note if anyone receives the extended time and how it is administered (i.e., others dismissed or wait), may need to ask 

TA if any students have accommodations in regular session (e.g., text to speech, magnify), note if text to speech with external 

technology and if any issues or difficulties or if worked well with the human reader, particularly with math. Student(s) comfortable with 
54 human reader or tools they are provided? 


Notes: Observers use accommodation job aid while observing separate session, note descrepancies. 


Post-Assessment Activities 


Collect student test booklet and any test materials. (10a-d) If any item is 
missed, rate 3-NM and list item(s) missed in box to right. 


60 Additional Comments/Notes: 
61 
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Appendix B — Debriefing Protocol for Test Administration Site Visits 


Test Administrator and/or Coordinator Debriefing Protocol for CBT 


PARCC Operational Testing 2015 
PBA or EOY Test Administrations 


Test Coordinator and/or Administrator Debriefing Protocol - Computer Based Test 


(Bold = priority questions) 


1 Describe how the computer-based test (CBT)went today? 
1a Follow up: If responds positive or negative, ask in what ways. 


1b Follow up: If problems or issues occurred, ask about them specifically. (Probes: graphics/sound issues, 
connectivity, students can answer as taught [answer length, no improper fractions] 
1b 


1c Follow up: Did testing today go better, same, or worse than other PARCC test administrations you have 
recently done? 


1c 

2 Was student engagement high, moderate, or low? Ask them to explain their rating. 
2 

3 What were some of the questions students asked during the assessment? 

3 


4 Was the test administration manual beneficial in preparing for the CBT? If so, or if not, ask in what 
ways. (Probes: was it well organized, too much info (what wasn't needed), hard to find what you need) 
4 


5 Were the scripts easy to follow and provide the right amount of information for students? If not, how 
would you improve them? 
5 


6 Did you receive any training in addition to the administration manual? Describe what that entailed. 
(Probes: did you watch videos, did you practice, focus is on PARCC training, not district) 
6 


7 Did all students (SD and ELL, too) access the practice tests, sample items, and tutorials prior to taking 
the CBT? If so, how long ago and do you think they helped prepare the student for CBT? If not, what 
were reasons they didn't receive the CBT training? 


8 Have you called the Support Center for any issue? If so, describe the issue(s). (Probes: did you get a 
satisfactory answer, was the time it took reasonable, did you have to call back) 
8 


9 Ifyou observed a separate accommodation session, ask if the TA received any specific training, 
particularly for Human Reader? 
9 


10 Does PARCC testing require more time and resources to administer than previous state tests you 


administered? If so, is it just because its new or can you provide specific examples. (a list is fine) 
10 


11 Are there any additional comments you would like to provide regarding test administration, testing 
materials, training materials, or support? 
11 
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Test Administrator and/or Coordinator Debriefing Protocol for PBT 


PARCC Operational Testing 2015 
PBA or EOY Test Administrations 


Test Coordinator and/or Administrator Debriefing Protocol - Paper Based Test 


(Bold = priority questions) 


1 Describe how the test went today? 


1a Follow up: If responds positive or negative, ask in what ways. 
1b Follow up: If problems or issues occurred, ask about them specifically. 
1b 


1c Follow up: Did the test today go better, the same, or worse than other PARCC test administrations you 


2 Was student engagement high, moderate, or low? Ask them to explain their rating. 
2 

3 What were some of the questions students asked during the assessment? 

3 


4 Was the test administration manual beneficial in preparing for the test? If so, or if not, ask in what 
ways. (Probes: was it well organized, too much info (what wasn't needed), hard to find what you 


Were the scripts easy to follow and provide the right amount of information for students? If not, how 
would you improve them? 


Did you receive any training in addition to the administration manual? Describe what that entailed. 
(Probes: did you watch videos, focus is on PARCC training, not district) 


Have you called the Support Center for any issue? If so, describe the issue(s). (Probes: did you get a 
satisfactory answer, was the time it took reasonable, did you have to call back) 


If you observed a separate accommodation session, ask if the TA received any specific training? 


Does PARCC testing require more time and resources to administer than previous state tests you 
administered? If so, is it just because its new or can you provide specific examples. (a list is fine) 


Are there any additional comments you would like to provide regarding test administration, testing 
materials, training materials, or support? 
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Appendix C — Test Administrator Survey Items (and codebook) for 
Computer/Tablet-based Test Administration 


PARCC Test Administrator Survey: Computer-based Test Administration 


Thank you for administering the PARCC Test. Now, we want to ask you some questions about 
the assessment(s) you just administered. We will use your answers to help improve the test 
administration process. 


(Background) 


1. Please identify which type of assessment you most recently administered: 
Variable Name: CBT_Assmnt_Type_S15 


o Performance-based assessment (PBA) Value = 1 
o End-of-year assessment (EOY) Value = 2 


ALL OF YOUR RESPONSES TO THE ITEMS ON THIS SURVEY SHOULD BE IN REFERENCE TO THE 
ASSESSMENT TYPE YOU JUST ADMINISTERED (I.E., WHICH EVER OPTION YOU SELECTED FOR 
QUESTION 1). 


2. Please identify the location where you administered the assessment. 
Variable Name: CBT_Location_S15 


AR Value = 1 
CO Value = 2 
DC Value = 3 
IL Value = 4 
MA Value =5 
MD Value =6 
MS Value = 7 
NJ Value = 8 
NM Value =9 
OH Value = 10 
RI Value = 11 


Oo 000802C00~20UC~UMDNCOCO COON 
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Please enter the code and name of the LEA/District in which you administered the 
assessment./4 


2a. LEA/District Code: ___ Variable Name: CBT_LEA_DistrictCode_txt_S15 
2b. District Name: Variable Name: CBT_DistrictName_txt_S15 


Please enter the code and name of the school in which you administered the assessment. 


2c. School Code: Variable Name: CBT_SchoolCode_txt_S15 
2d. School Name: Variable Name: CBT_SchoolName_txt_S15 


3. Please indicate which assessment(s) you administered for English Language Arts/Literacy 
(ELA). Select all that apply 


Not Applicable Variable Name: CBT_ELA_NA_S15; Value = 1 
ELA/L Grade 3 Variable Name: CBT_ELA_G3_S15; Value = 1 
ELA/L Grade 4 Variable Name: CBT_ELA_G4_S15; Value = 1 
ELA/L Grade 5 Variable Name: CBT_ELA_G5_S$15; Value = 1 
ELA/L Grade 6 Variable Name: CBT_ELA_G6_S$15; Value = 1 
ELA/L Grade 7 Variable Name: CBT_ELA_G7_S15; Value = 1 
ELA/L Grade 8 Variable Name: CBT_ELA_G8_S$15; Value = 1 
ELA/L Grade 9 Variable Name: CBT_ELA_G9_S$15; Value = 1 
ELA/L Grade 10 Variable Name: CBT_ELA_G10_S15; Value = 1 
ELA/L Grade 11 Variable Name: CBT_ELA_G11_S15; Value = 1 


o0o0o0ClmOlOUmUmUCOOWUCUCOUCOUCUCOOUCOOUlCO 


4. Please indicate which assessment(s) you administered for Mathematics. Select all that apply 


Not Applicable Variable Name: CBT_M_NA_S15; Value = 1 

Math Grade 3 Variable Name: CBT_M_G3_S$15; Value = 1 

Math Grade 4 Variable Name: CBT_M_G4_S$15; Value = 1 

Math Grade 5 Variable Name: CBT_M_G5_ $15; Value = 1 

Math Grade 6 Variable Name: CBT_M_G6_S$15; Value = 1 

Math Grade 7 Variable Name: CBT_M_G7_S15; Value = 1 

Math Grade 8 Variable Name: CBT_M_G8_S15; Value = 1 
Algebra 1 Variable Name: CBT_M_ALG1_S15; Value = 1 
Geometry Variable Name: CBT_M_GEO_S15; Value = 1 

Algebra 2 Variable Name: CBT_M_ALG2_S15; Value = 1 
Integrated Math 1 Variable Name: CBT_M_INT1_S$15; Value = 1 
Integrated Math 2 Variable Name: CBT_M_INT2_S15; Value = 1 
Integrated Math 3 Variable Name: CBT_M_INT3_S15; Value = 1 


Oo O0O0080C00260C~CUL0C—CUm0DUCOOOOOOOO CO 


These state were presented with questions 2a — 2d: DC, MD, MS, OH, RI. These states were not presented with 
questions 2a — 2d: AR, CO, IL, MA, NJ, NM. 
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(Test Administrator’s Training Experience) 


1. Have you ever administered a computer-based test to students before (either at the state or 
district level)? Variable Name: CBT_PrevCBTAdmin_S15 

o Yes Value = 1 

o No Value = 2 


2. Please indicate your level of agreement with the following statement, “This training 
effectively prepared me to administer the PARCC assessments.” 


Strongly Strongly Not 
Disagree | Disagree Agree Agree Applicable 
Value =2 | Value =3 Value = 4 Value =5 


Accessibility Features & Accommodations 
Training Module CBT_AFAModule_S15 


Administration of Computer-Based Assessments 
for Test Administrators Training Module 
CBT_AdminModule_S15 


Infrastructure Trials: Running a Dress Rehearsal 
Training Module CBT_InfastructWodule_S15 


Introduction to PARCC Training Modules 
Training Module CBT_IntroPARCCModule_S15 


PearsonAccess"™ Training Module 
CBT_PearsonAccModule_S15 


Personal Needs Profile (PNP) Training Module 
CBT_PNPModule_S15 


Proctor Caching & TestNav Configuration 
Training Module 
CBT_ProcCachTestNavModule_S15 


Student Readiness Resources for PARCC Training 
Module CBT_StudReadResModule_S15 


Technology “Readiness” for Schools & Districts 
Training Module 
CBT_TechReadinessModule_S15 


Training from my state department of education 
CBT_State_Training_$15 


Training from my school 
CBT_School_Training_S15 
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3. The PARCC online training(s) prepared me to resolve basic problems related to technology 
(e.g., logging students in, exiting the test, etc.) 


Variable Name: CBT_Mod_FixTechProb_S15 
o Strongly Disagree Value = 1 

Disagree Value = 2 

Agree Value = 3 

Strongly Agree Value = 4 

Not applicable Value =5 


Oo 0 0 90 


4. Do you have any specific recommendations on how PARCC can improve its online training? 
Variable Name: CBT_ModRecommendations_txt_S15 


(Test Administration Experience) 
1. Please indicate your level of agreement with each of the following statements. 


a. The policies and procedures within the Test Administrator Manual were easy for me to understand. 
Variable Name: CBT_Pol_ProcUnderstand_S15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value =3 

o Strongly Agree Value = 4 


b. The instructions (including scripts for administering the assessment) within the Test Administrator 
Manual were easy for me to implement. Variable Name: CBT_InstructionEaselmp_S15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value = 3 

o Strongly Agree Value = 4 


c. Students appeared to understand the directions | read to them during the test administration. 
Variable Name: CBT_StudentUnderstand_S15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value =3 

o Strongly Agree Value = 4 


d. The instructions | read to the students covered all of the information necessary to take the test. 
Variable Name: CBT_Necessary_Info_$15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value =3 

o Strongly Agree Value = 4 
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2. Please indicate if student(s) asked questions about the following topics. Select all that apply 


O 


Oo 000 0 0 


Clarification on the instructions | read to them Variable: CBT_StudQs_TAinstr_S15 

What to do if there were technology-related problems Variable: 
CBT_StudQs_TechProb_$15; Value=1 

How to find or use tools Variable: CBT_StudQs_Tools_S15; Value=1 

How to navigate through the test Variable: CBT_StudQs_Navigate_S15; Value=1 

How to mark answers and enter responses Variable: CBT_StudQs_MarkAns_ $15; Value = 1 
How to exit the test when finished Variable: CBT_StudQs_Exit_$15; Value=1 

What to do after they completed the test Variable: CBT_StudQs_AfterCmplt_S15; Value=1 
Other, (please specify). Variable: CBT_StudQs_Other_S15; Value=1 

Other: Variable: CBT_StudQs_Other_txt_S15 


3. Please indicate if any of the following technology-related problems occurred during testing. 


Be sure to respond based on the type of assessment you most recently administered (i.e., 
either PBA or EOY). Select all that apply 


Oo O00 0 0 


O 


Other: 


No technology-related problems occurred. Variable: CBT_NoProb_Occur_S15; Value=1 
Students had difficulty logging into the assessment Variable: CBT_LogOn_Occur_S15; Value=1 
Device(s) stopped working Variable: CBT_DevStop_Occur_S15; Value=1 

Devices worked slowly Variable: CBT_DevSlow_Occur_S15; Value=1 

Lost internet connectivity Variable: CBT_LostConnect_Occur_S15; Value=1 

System disconnected or logged students out of assessment during administration 
Variable: CBT_SysDiscnct_Occur_S15; Value = 1 

TestNav online tools (e.g., highlighter, calculator) did not work appropriately Variable: 
CBT_TestNav_Occur_S15; Value = 1 

Accessibility features (e.g., eliminate answer) did not work appropriately. Variable: 
CBT_AccFeatures_Occur_S15; Value = 1 

Other, (please specify). Variable: CBT_OtherProb_Occur_S15; Value=1 


Variable: CBT_OtherProb_Occur_txt_$15 


4. Did your students have sufficient time to finish the test? Variable Name: CBT_Suff_time_S15 


Most students finished very early. Value = 1 
Most students finished on time. Value = 2 
Most students had to rush to finish. Va/ue = 3 
Most students did not finish. Value = 4 
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PARCC 


(Accessibility Features and Accommodations) 


1. Did you read the “PARCC Accessibility Features and Accommodations Manual” prior to 
administration? Variable Name: CBT_AFA_Manual_Read_$15 


o Yes Value=1 
o No Value=2 


2. Please indicate your level of agreement with the following statement: This particular 
item/component effectively informed me of how to administer accessibility features and 
accommodations on the PARCC assessments. 


Strongly 
Disagree 
Value=1 


Disagree 
Value=2 


Agree 
Value=3 


Strongly 
Agree 
Value=4 


Not 
Applicable 
Value=5 


a. PARCC Accessibility Features and 
Accommodations Manual — 3"¢ 
Edition 
CBT_AFA_Manual_effective_$15 


b. Accessibility and 
Accommodations Training Module 
CBT_AA_TrainMod_effective_S15 


c. Personal Needs Profile Training 
Module 
CBT_PNP_TrainMod_effective_S15 


d. Personal Needs Profile Field 
Definitions 
CBT_PNP_Defs_effective_S15 


e. Training or resources from my 
state department of education 
CBT_AFA_state_effective_S15 


f. Training from my district 
CBT_AFA_dist_effective_S15 


g. Training from my school 
CBT_AFA_school_effective_S15 
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3. Did you administer the following accessibility features or accommodations to any of your 


students? Select all that apply 


o Human reader for mathematics CBT_humanread_M_S15; value = 1 


O00 0 0 


Human reader for ELA/L CBT_humanread_ELA_S15; value = 1 
Human scribe CBT_humanscribe_S15; value = 1 
Human signer CBT_humansign_$15; value = 1 
General test directions read aloud in students’ native languages 
CBT_humanread_drctn_S$15; value = 1 

External speech-to-text device CBT_ext_texttospch_S15; value = 1 
Extended time CBT_extendtime_S15; value = 1 
External word prediction device CBT_ext_wrdprddev_S15; value = 1 
Calculator or calculation device on non-calculator section; CBT_calculator_S15; value = 1 
Other external assistive technology devices CBT_otherextdev_$15; value = 1 


4. Please indicate your level of agreement with the following statement: This particular 
appendix/guideline effectively informed me of how to administer accessibility features and 
accommodations on the PARCC assessments. 


Strongly 
Disagree 
Value=1 


Disagree 
Value=2 


Agree 
Value=3 


Strongly 
Agree 
Value=4 


Did Not Administer 
Relevant 
Accommodation/ 
Accessibility Feature 
Value=5 


a. Appendix A: Accessibility Features and 
Accommodations for Students taking the 
Paper-Based PARCC Assessments 
CBT_AFA_AppA_S$15 


b. Appendix B: Test Administration 
Protocol for the Human Reader 
Accommodations for ELA/L and the 
Human Reader Accessibility Feature for 
Math 

CBT_AFA_AppB_S15 


c. Appendix C: Protocol for the Use of the 
Scribe and for Transcribing Student 
Responses 

CBT_AFA_AppC_S15 


d. Appendix E: Guidance for Selecting and 
Administering the Extended Time 
Accommodation 

CBT_AFA_AppE_S15 


e. Appendix |: PARCC ELA Audio 
Guidelines 
CBT_AFA_Appl_515 


f. Appendix J: PARCC Math Audio 
Guidelines 
CBT_AFA_AppJ S15 


g. Assistive Technology Guidelines 
CBT_AFA_AsstTech_S15 


h. PARCC Technical Assistance Bulletin — 
PARCC Assessments and Students with 
Visual Impairment, Including Blindness 
CBT_AFA_TechBulletin_S15 
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5. The process for identifying accessibility features and accommodations in advance (via the 
Personal Needs Profile and associated trainings) was clear and easy to follow. 
CBT_PNP_clear_S15 

e Strongly Disagree Va/ue=1 

e Disagree Value=2 

e Agree Value=3 

e Strongly Agree Value=4 

e Not Applicable. | was not responsible for uploading and/or adding PNP data. Value=5 


6. Were all the accommodations/accessibility features that were pre-identified in the PNP made 
available to students during a practice session with sample items, tutorial(s), or practice test(s)? 
Variable Name: CBT_AFA_available_S15 

e Yes Value =1 

e No Value = 2 

e Not Sure/Don’t Know Value = 3 

e NotApplicable Value =4 


7. During administration, were changes made to a student’s PNP or to the availability of 
accommodations or accessibility features? CBT_PNP_changes_S15 

e Yes Value=1 

e No Value=2 

e Not Sure/Don’t Know Value=3 

e Not Applicable Value = 4 


(Sample Items/Tutorials) 


1. How did students in your session(s) practice with PARCC content prior to administration? Select all 
that apply. 
o Mystudents practiced with sample items prior to administration CBT_sample_items_S15; Value 
=i 
o Mystudents practiced with tutorials prior to administration CBT_tutorials_S15; Value = 1 
My students took a practice test prior to administration CBT_practice_test_S15; Value = 1 
o Mystudents did not practice with PARCC content prior to administration CBT_no_practice_S15; 
Value = 1 
o Not sure/Don’t know CBT_pract_dontknow_S15; value = 1 


(e) 


2. How did you, as a Test Administrator, work with PARCC content prior to administration? 
Select all that apply. 
o | reviewed the sample items prior to administration CBT_TA_sample_items_S15; Value = 1 
o | reviewed the tutorials prior to administration CBT_TA_tutorials_S15; Value = 1 
o |took a practice test prior to administration CBT_TA_practice_test_S15; Value = 1 
o  Idid not practice with PARCC content prior to administration CBT_TA_no_practice_S15; Value =1 
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3. Completion of the PARCC tutorial helped me to better understand the tools and 
functionalities of the TestNav system. 
Variable Name: CBT_TATutorial_UndrstndTestNav_S15 


oO 


Oo 0 0 0 


Strongly Disagree Value=1 
Disagree Value=2 

Agree Value=3 

Strongly Agree Value=4 
Not applicable Va/ue=5 


(Final Feedback) 


1. Please use the space below to provide any other recommendations (not already provided in 
the questions above) that may help PARCC improve its administration policies and procedures 
(e.g., administration instructions, security policies/procedures, and/or the role of the Test 


Administrator). 
Variable Name: CBT_FinalFeedback_txt_S15 
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Appendix D — Test Administrator Survey Items (and codebook) for Paper-based Test 
Administration 


PARCC Test Administrator Survey: Paper-based Test Administration 


Thank you for administering the PARCC Test. Please note that you should only respond to this 
survey once after you have completed administering all of your PBA sessions and once after you 
have completed all of your EOY sessions. Now, we want to ask you some questions about the 
assessment(s) you just administered. We will use your answers to help improve the test 
administration process. 


(Background) 


1. Please identify which type of assessment you most recently administered: 
Variable Name: PBT_Assmnt_Type_S15 


o Performance-based assessment (PBA) Value = 1 
o End-of-year assessment (EOY) Value = 2 


ALL OF YOUR RESPONSES TO THE ITEMS ON THIS SURVEY SHOULD BE IN REFERENCE TO THE 
ASSESSMENT TYPE YOU JUST ADMINISTERED (I.E., WHICH EVER OPTION YOU SELECTED FOR 
QUESTION 1). 


2. Please identify the location where you administered the assessment. 

Variable Name: PBT_Location_S15 
o AR Value =1 

CO Value = 2 

DC Value = 3 

IL Value = 4 

MA Value = 5 

MD Value =6 

MS Value = 7 

NJ Value =8 

NM Value = 9 

OH Value = 10 

RI Value = 11 


C0CO0O00Cm0dUCOlOUCOUCUCOOUUCOOUlLCO 
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Please enter the code and name of the LEA/District in which you administered the 


assessment.!3 


2a. LEA/District Code: 


2b. District Name: 


Quality of Test Administration 


Variable Name: PBT_LEA_DistrictCode_txt_S15 


___ Variable Name: PBT_DistrictName_txt_S15 


Please enter the code and name of the school in which you administered the assessment. 


2c. School Code: 


2d. School Name: 


__ Variable Name: PBT_SchoolCode_txt_S15 


__ Variable Name: PBT_SchoolName_txt_S15 


3. Please indicate which assessment(s) you administered for English Language Arts/Literacy 
(ELA/L). Select all that apply 
Not Applicable Variable Name: PBT_ELA_NA_S15; Value = 1 


ELA/L Grade 3 Variable Name: 
ELA/L Grade 4 Variable Name: 
ELA/L Grade 5 Variable Name: 
ELA/L Grade 6 Variable Name: 
ELA/L Grade 7 Variable Name: 
ELA/L Grade 8 Variable Name: 


PBT_ELA_G3_S15; Value = 1 
PBT_ELA_G4_S15; Value = 1 
PBT_ELA_G5_S15; Value = 1 
PBT_ELA_G6_S15; Value = 1 
PBT_ELA_G7_S15; Value = 1 
PBT_ELA_G8_S15; Value = 1 


ELA/L Grade 9 Variable Name: PBT_ELA_G9_ S15; Value = 1 
ELA/L Grade 10 Variable Name: PBT_ELA_G10_$15; Value = 1 
ELA/L Grade 11 Variable Name: PBT_ELA_G11_$15; Value = 1 


o0o0o0ClmUOdUWUmUmUCOUCUCOUCOUCOOUCOOUlCO 


4. Please indicate which assessment(s) you administered for Mathematics. Select all that apply 
o Not Applicable Variable Name: PBT_M_NA_S15; Value = 1 

Math Grade 3 Variable Name: PBT_M_G3_S15; Value = 1 

Math Grade 4 Variable Name: PBT_M_G4_S15; Value = 1 

Math Grade 5 Variable Name: PBT_M_G5_S15; Value = 1 

Math Grade 6 Variable Name: PBT_M_G6_S15; Value = 1 

Math Grade 7 Variable Name: PBT_M_G7_S15; Value = 1 

Math Grade 8 Variable Name: PBT_M_G8_S15; Value = 1 

Algebra 1 Variable Name: PBT_M_ALG1_ $15; Value = 1 

Geometry Variable Name: PBT_M_GEO_S15; Value = 1 

Algebra 2 Variable Name: PBT_M_ALG1_ $15; Value = 1 

Integrated Math 1 Variable Name: PBT_M_INT1_S15; Value = 1 

Integrated Math 2 Variable Name: PBT_M_INT2_S15; Value = 1 

Integrated Math 3 Variable Name: PBT_M_INT3_S15; Value = 1 


oO O0000C0UCCU0OUCUCOOUCUCOUCUCOOWUCUCOOLCOO 


These state were presented with questions 2a — 2d: DC, MD, MS, OH, RI. These states were not presented with 
questions 2a — 2d: AR, CO, IL, MA, NJ, NM. 
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(Test Administrator’s Training Experience) 


1. Please indicate your level of agreement with the following statement: This training effectively 
prepared me to administer the PARCC assessments. 


Strongly Strongly Not 
Disagree | Disagree Agree Applicable 
Value = 3} Value=4 | Value=5 


Introduction to PARCC Training Modules 
PBT_IntroPARCCModule_S15 


Accessibility Features & Accommodations 
Training Module PBT_AFAModule_S15 


Administration of Paper-Based 
Assessments for Test Administrators 
Training Module PBT_AdminModule_S15 


PearsonAccess™' Training Module 
PBT_PearsonAccModule_S15 


Student Readiness Resources for PARCC 
Training Module 
PBT_StudReadResModule_S15 


Training from my state department of 
education PBT_State_Training_ S15 


Training from my district 
PBT_Dist_Training_S15 


Training from my school 
PBT_School_Training $15 


Student Registration Import Training 
Module PBT_StudReglmpModule_S15 


2. Do you have any specific recommendations on how PARCC can improve its online training? 
Variable Name: PBT_ModRecommendations_txt_S15 
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(Test Administration Experience) 
1. Please indicate your level of agreement with each of the following statements. 


a. The policies and procedures within the Test Administrator Manual were easy for me to 
understand. 
Variable Name: PBT_Pol_ProcUnderstand_S15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value =3 

o Strongly Agree Value = 4 


b. The instructions (including scripts for administering the assessment) within the Test 
Administrator Manual were easy for me to implement. 
Variable Name: PBT_InstructionEaselmp_ $15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value =3 

o Strongly Agree Value = 4 


c. Students appeared to understand the directions | read to them during the test 
administration. 
Variable Name: PBT_StudentUnderstand_S15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value = 3 

o Strongly Agree Value = 4 


d. The instructions | read to the students covered all of the information necessary to take the 
test. 
Variable Name: PBT_Necessary_Info_S15 

o Strongly Disagree Value = 1 

o Disagree Value = 2 

o Agree Value = 3 

o Strongly Agree Value = 4 
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2. Please indicate if student(s) asked questions about the following topics. Select all that apply. 


o Clarification on the instructions | read to them Variable: PBT_StudQs_TAinstr_S15; Value=1 
o When they could use a calculator Variable: PBT_StudQs_Calculator_S15; Value=1 

o Howto mark answers and enter responses Variable: PBT_StudQs_MarkAns_S15; Value = 1 
o What to do after they completed the test Variable: PBT_StudQs_AfterCmplt_S15; Value=1 
o Other, (please specify). Variable: PBT_StudQs_Other_S15; Value=1 


Other: Variable: PBT_StudQs_Other_txt_$15 
3. Did your students have sufficient time to finish the test? Variable Name: PBT_Suff_time_S15 


e Most students finished very early. Value = 1 

e Most students finished on time. Value = 2 

e Most students had to rush to finish. Va/ue = 3 
e Most students did not finish. Value = 4 
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1. Did you read the “PARCC Accessibility Features and Accommodations Manual” prior to 
administration? Name: PBT_AFA_Manual_Read_S15 


o Yes Value=1 
o No Value=2 


2. For each item below, indicate your level of agreement with the following statement: This 
particular item/component effectively informed me of how to administer accessibility features 


and accommodations on the PARCC assessments. 


Strongly 
Disagree 
Value=1 


Disagree 
Value=2 


Agree 
Value=3 


Strongly 
Agree 
Value=4 


Not 
Applicable 
Value=5 


a. PARCC Accessibility Features and 
Accommodations Manual — 3 
Edition 
PBT_AFA_Manual_effective_S15 


b. Accessibility and 
Accommodations Training Module 
PBT_AA_TrainMod_effective_S15 


c. Personal Needs Profile Training 
Module 
PBT_PNP_TrainMod_effective_S15 


d. Personal Needs Profile Field 
Definitions 
PBT_PNP_Defs_effective_S15 


e. Training or resources from my 
state department of education 
PBT_AFA_state_effective_S15 


f. Training from my district 
PBT_AFA_dist_effective_S15 


g. Training from my school 
PBT_AFA_school_effective_S15 
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3. Did you administer the following accessibility features or accommodations to any of your 
students? Select all that apply. 


Human reader for mathematics PBT_humanread_M_S15; value = 1 

Human reader for ELA/L PBT_humanread_ELA_S15; value = 1 

Human scribe PBT_humanscribe_S15; value = 1 

Human signer PBT_humansign_S15; value = 1 

General test directions read aloud in students’ native languages 
PBT_humanread_drctn_S15; value = 1 

External speech-to-text device PBT_ext_texttospch_S15; value = 1 

Extended time PBT_extendtime_5S15; value = 1 

External word prediction device PBT_ext_wrdprddev_S15; value = 1 
Calculator or calculation device on non-calculator sections; PBT_calculator_S15; value = 1 
Other external Assistive Technology devices PBT_otherextdev_S15; value = 1 


00000 


00000 


4. For each item below, indicate your level of agreement with the following statement: This 
particular appendix/guideline effectively informed me of how to administer accessibility 
features and accommodations on the PARCC assessments. 


Did Not Administer 


Relevant 
Strongly Strongly Accommodation/ 
Disagree | Disagree Agree Agree Accessibility Feature 
Value=1 Value=2 | Value=3 | Value=4 Value=5 


a. Appendix A: Accessibility Features and 
Accommodations for Students taking the 
Paper-Based PARCC Assessments 
PBT_AFA_AppA_S15 


b. Appendix B: Test Administration Protocol 
for the Human Reader Accommodations for 
ELA/L and the Human Reader Accessibility 
Feature for Math 

PBT_AFA_AppB_ S15 


c. Appendix C: Protocol for the Use of the 
Scribe and for Transcribing Student 
Responses 

PBT_AFA_AppC_S15 


d. Appendix E: Guidance for Selecting and 
Administering the Extended Time 
Accommodation 

PBT_AFA_AppE_S15 


e. Appendix |: PARCC ELA Audio Guidelines 
PBT_AFA_Appl_515 


f. Appendix J: PARCC Math Audio 
Guidelines 
PBT_AFA_AppJ_S15 


g. Assistive Technology Guidelines 
PBT_AFA_AsstTech_S15 


h. PARCC Technical Assistance Bulletin — 
PARCC Assessments and Students with 
Visual Impairment, Including Blindness 
PBT_AFA_TechBulletin_S15 
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5. The process for identifying accessibility features and accommodations in advance (via the 
Personal Needs Profile and associated trainings) was clear and easy to follow. 
PBT_PNP_clear_S15 


e Strongly Disagree Value=1 

e Disagree Value=2 

e Agree Value=3 

e Strongly Agree Value=4 

e Not Applicable. | was not responsible for uploading and/or adding PNP data. Value=5 


6. Were all the accommodations/accessibility features that were pre-identified in the PNP made 
available to students during a practice session with sample items, tutorial(s), or practice test(s)? 
Variable Name: PBT_AFA_available_S15 

e Yes Value = 1 

e No Value =2 

e Not Sure/Don’t Know Value = 3 

e Not Applicable Value = 4 


7. During administration, were changes made to a student’s PNP or to the availability of 
accommodations or accessibility features? PBT_PNP_changes S15 

e Yes Value=1 

e No Value=2 

e Not Sure/Don’t Know Value=3 

e Not Applicable Value = 4 
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(Sample Items/Tutorials) 


1. How did students in your session(s) practice with PARCC content prior to administration? 
Select all that apply. 


o Mystudents practiced with sample items prior to administration 
PBT_sample_items_$15; Value = 1 

o Mystudents practiced with tutorials prior to administration PBT_tutorials_S15; Value = 1 
My students took a practice test prior to administration PBT_practice_test_$15; Value = 1 
My students did not practice with PARCC content prior to administration 
PBT_no_practice_S15; Value = 1 

© Not sure/Don’t know PBT_pract_dontknow_$15; value = 1 


2. How did you, as a Test Administrator, work with PARCC content prior to administration? 
Select all that apply. 


o lreviewed the sample items prior to administration PBT_TA_sample_items_S15; Value 
=1 
| reviewed the tutorials prior to administration PBT_TA_tutorials_$15; Value = 1 
| took a practice test prior to administration PBT_TA_practice_test_S15; Value = 1 
| did not practice with PARCC content prior to administration PBT_TA_no_practice_S15; 
Value = 1 


3. Completion of the PARCC paper-based student tutorials helped better prepare me for 
administering the assessment. Variable Name: PBT_TAPracltems_PrepAdmin_S15 


Strongly Disagree Va/ue=1 
Disagree Value=2 

Agree Value=3 

Strongly Agree Value=4 
Not applicable Va/ue=5 


Oo O00 0 0 


(Final Feedback) 


1. Please use the space below to provide any other recommendations (not already provided in 
the questions above) that may help PARCC improve its administration policies and procedures 
(e.g., administration instructions, security policies/procedures, and/or the role of the Test 


Administrator). 
Variable Name: PBT_FinalFeedback_txt_S15 
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Appendix E — Student Survey Items (and Code Book) for 
CBT EOY Mathematics Assessment 


Now that you have finished the PARCC mathematics test, we would like to know about your 
experience. We will use your responses to help make a better testing experience for students in 
the future. 


Directions: 


Please answer the questions below as best that you can. If you do not understand a question, 
please skip it and go on. 


1. Did you understand all of the directions read by the person who gave you the test? 
Variable Name = CBT_Math_Understand_Instr 

Yes Value = 1 

No Value = 2 


2. How often was it hard to understand the directions for the questions on this test? 
Variable Name = CBT_Math_Difficult_Instr 

Almost Always Value = 1 

Most of the time Value = 2 

Some of the time Value = 3 

Almost Never Value = 4 


3. How many questions asked you about things you have not learned in school this year? 
Variable Name = CBT_Math_Qs_Not_Learned 

All of them Value = 1 

Most of them Value = 2 

Few of them Value = 3 

None of them Value = 4 


4. How difficult was this test? 

Variable Name = CBT_Math_DifTest 
It was easier than my school work Value = 1 
It was about the same as my school work Value = 2 
It was harder than my school work Value = 3 


5. Did you have enough time to finish this test? 
Variable Name = CBT_Math_Time_Finish 

| finished very early Value = 1 

| finished on time Value = 2 

| had to rush to finish Value = 3 

| did not finish Value = 4 
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6. How often do you use a computer or tablet at home? 
Variable Name = CBT_Math_Home_Tech 

Every day Value = 1 

A few times a week Value = 2 

A couple of times a month or less Value = 3 

| don’t have a computer or a tablet at home Value = 4 


7. How often do you use a computer or tablet in school? 
Variable Name = CBT_Math_School_Tech 
Every day Value = 1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 
| have never used a computer or tablet in school Value = 4 


8. When writing a story or essay, how often do you use a computer or tablet? 
Variable Name = CBT_Math_Tech_Writing 

All of the time Value = 1 

Most of the time Value = 2 

Some of the time Value = 3 

Never Value = 4 


9. Would you rather take this test on paper OR on a computer or tablet? 
Variable Name = CBT_Math_Test_Pref 

On paper Value = 1 

On a computer or tablet Value = 2 


10. What did you use to take this test? 
Variable Name = CBT_Math_TechType 
Desktop computer Value = 1 
Laptop computer Value = 2 
Tablet with a keyboard Value = 3 
Tablet without a keyboard Value = 4 


11. Was it easy to type your answers? 
Variable Name = CBT_Math_Type 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not type any answers Value = 3 


12. How many times did you practice on a computer or tablet to get ready for this test? 
Variable Name = CBT_Math_TechPractice 

Never Value = 1 

Once Value = 2 

More than once Value = 3 
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13. Was it easy to use the highlighter? 
Variable Name = CBT_Math_Highlight 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not use the highlighter Value = 3 


14. Was it easy to make pictures or words bigger or smaller? 
Variable Name = CBT_Math_Change_Size 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not change the size of pictures or words Value = 3 


15. Was it easy to enter math symbols and numbers for your answers? 
Variable Name = CBT_Math_Enter_SymNum 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not enter math symbols or numbers Value = 3 


16. Was it easy to use the calculator? 
Variable Name = CBT_Math_Calc 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not use the calculator Value = 3 


17. Did you have problems logging into the test? 
Variable Name = CBT_Math_Comp_Log 

Yes Value = 1 

No Value =2 


18. Did the computer (or tablet) stop working? 
Variable Name = CBT_Math_Comp_Stop 

Yes Value = 1 

No Value =2 


19. Did the computer (or tablet) work slowly? 
Variable Name = CBT_Math_Comp_Slow 

Yes Value = 1 

No Value =2 


20. Did you have a hard time dragging or moving things on the screen? 
Variable Name = CBT_Math_Move_Objects 

Yes Value = 1 

No Value =2 
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21. Did you have a hard time making changes to your answers? 
Variable Name = CBT_Math_Difficult_Change_Ans 

Yes Value = 1 

No Value =2 


22. Please click on the sentence that is most true: 

Variable Name = CBT_Math_Probs 

There were NO problems with the computer (or tablet) while | was taking the test. Va/ue=1 
There were problems with the computer (or tablet) while | was taking the test. Va/ue =2 


23. Please use the space below to tell us what you liked and did not like about the test. 
Variable Name = CBT_Math_Feedback_txt 
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Appendix F — Student Survey Items (and Code Book) for 
CBT EOY ELA Assessment 


Student Survey Items (and Code Book) for Computer/Tablet-based PARCC ELA/L Tests 

Now that you have finished the PARCC ELA/L test, we would like to know about your 
experience. We will use your responses to help make a better testing experience for students in 
the future. 


Directions: 
Please answer the questions below as best that you can. If you do not understand a question, 
please skip it and go on. 


1. Did you understand all of the directions read by the person who gave you the test? 
Variable Name = CBT_ELA_Understand_Instr 

Yes Value = 1 

No Value = 2 


2. How often was it hard to understand the directions for the questions on this test? 
Variable Name = CBT_ELA_Difficult_Instr 

Almost Always Value = 1 

Most of the time Value = 2 

Some of the time Value = 3 

Almost Never Value = 4 


3. How many questions asked you about things you have not learned in school this year? 
Variable Name = CBT_ELA_Qs_Not_Learned 

All of them Value = 1 

Most of them Value = 2 

Few of them Value = 3 

None of them Value = 4 


4. How difficult was this test? 

Variable Name = CBT_ELA_DifTest 
It was easier than my school work Value = 1 
It was about the same as my school work Value = 2 
It was harder than my school work Value = 3 


5. Did you have enough time to finish this test? 
Variable Name = CBT_ELA_Time_Finish 

| finished very early Value = 1 

| finished on time Value = 2 

| had to rush to finish Value = 3 

| did not finish Value = 4 
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6. How often do you use a computer or tablet at home? 
Variable Name = CBT_ELA_Home_Tech 

Every day Value = 1 

A few times a week Value = 2 

A couple of times a month or less Value = 3 

| don’t have a computer or a tablet at home Value = 4 


7. How often do you use a computer or tablet in school? 
Variable Name = CBT_ELA_School_Tech 
Every day Value = 1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 
| have never used a computer or tablet in school Value = 4 


8. When writing a story or essay, how often do you use a computer or tablet? 
Variable Name = CBT_ELA_Tech_Writing 

All of the time Value = 1 

Most of the time Value = 2 

Some of the time Value = 3 

Never Value = 4 


9. Would you rather take this test on paper OR on a computer or tablet? 
Variable Name = CBT_ELA_Test_Pref 

On paper Value = 1 

On a computer or tablet Value = 2 


10. What did you use to take this test? 
Variable Name = CBT_ELA_TechType 
Desktop computer Value = 1 
Laptop computer Value = 2 
Tablet with a keyboard Value = 3 
Tablet without a keyboard Value = 4 


11. Was it easy to type your answers? 
Variable Name = CBT_ELA_Type 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not type any answers Value = 3 


12. How many times did you practice on a computer or tablet to get ready for this test? 
Variable Name = CBT_ELA_TechPractice 

Never Value = 1 

Once Value = 2 

More than once Value = 3 
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13. Was it easy to use the highlighter? 
Variable Name = CBT_ELA_Highlight 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not use the highlighter Va/ue = 3 


14. Was it easy to make pictures or words bigger or smaller? 
Variable Name = CBT_ELA_Change_Size 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not change the size of pictures or words Value = 3 


15. Was it easy to move back and forth between passages or stories? 
Variable Name = CBT_ELA_Change_Pass 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not move between passages or stories Value = 3 


16. Was it easy to find information in the passages or stories when answering questions? 
Variable Name = CBT_ELA_Find_Info 

Yes, it was easy Value = 1 

No, it was hard Value = 2 

| did not move between stories or passages Value = 3 


17. Did you have problems logging into the test? 
Variable Name = CBT_ELA_Comp_Log_ 

Yes Value = 1 

No Value =2 


18. Did the computer (or tablet) stop working? 
Variable Name = CBT_ELA_Comp_Stop 

Yes Value = 1 

No Value =2 


19. Did the computer (or tablet) work slowly? 
Variable Name = CBT_ELA_Comp_Slow 

Yes Value = 1 

No Value =2 


20. Did you have a hard time dragging or moving things on the screen? 
Variable Name = CBT_ELA_Move_Objects 

Yes Value = 1 

No Value =2 
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21. Did you have a hard time making changes to your answers? 
Variable Name = CBT_ELA_Difficult_Change_Ans 

Yes Value = 1 

No Value =2 


22. Please click on the sentence that is most true: 

Variable Name = CBT_ELA_Probs 

There were NO problems with the computer (or tablet) while | was taking the test. Va/ue =1 
There were problems with the computer (or tablet) while | was taking the test. Value =2 


23. Please use the space below to tell us what you liked and did not like about the test. 
Variable Name = CBT_ELA_Feedback_txt 
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Appendix G — Item-level Results from Test Administrator Survey: 
CBT Administration with Breakouts for PBA and EOY 


Computer-Based 


Background Information 


Table G-1. TA Survey Respondents by Administration and State 


CBT PBA CBT EOY 
N-count % N-count % 
Arkansas 2,358 22.4 2,838 25.4 
Colorado 136 1.3 183 1.6 
District of Columbia 55 0.5 30 0.3 
Illinois 2,264 21.5 1,803 16.1 
Maryland 555 5.3 1,056 9.4 
Massachusetts 693 6.6 752 6.7 
Mississippi 92 0.9 72 0.6 
New Jersey 1,920 18.3 3647 32.6 
New Mexico 1256 12.0 330 2.9 
Ohio 463 4.4 433 3.9 
Rhode Island 718 6.8 44 0.4 
All States 10,545? 100.0 11,232° 100.0 


Note. Percentages are based on the total number of valid responses for each state. Number of 
missing for CBT PBA = 35. Number of missing for CBT EOY = 44. 
Includes the 35 cases that did not indicate their state. "Includes the 44 cases that did not indicate their state. 
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Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 N/A 

State n % n % n % n % n % n % n % n % n % n % 

Arkansas 426 181 421 179 392 166 393 16.7 412 17.5 397 168 335 142 319 135 129 55 135 5.7 
Colorado 26 19.1 28 20.6 28 20.6 24 17.6 25 18.4 22 16.2 25 18.4 32 235 21 15.4 7 5.1 
D.C. 18 32.7 19 34.5 11 20.0 6 10.9 4 7.3 4 7.3 -- -- 2 3.6 -- -- 4 7.3 
Illinois 502 22.2 482 21.3 467 20.6 $450 19.9 468 20.7 450 19.9 115 5.1 9 04 121 53 120 5.3 
Maryland 120 21.6 159 28.6 136 24.5 80 14.4 92 16.6 83 15.0 4 07 49 8.8 2 04 46 8.3 
Massachusetts 171 24.7 174 25.1 176 25.4 170 24.5 144 20.8 149 21.5 36 5.2 4 0.6 31 4.5 29 4.2 
Mississippi 29 31.5 17° «(18.5 7 7.6 12 13.0 10 10.9 9 9.8 - -- 13 14.1 1 1.1 17 18.5 
New Jersey 372 19.4 345 18.0 331 17.2 252 13.1 243 «12.7 219 11.4 232 12.1 241 #=+#12.6 206 107 143 7.4 
New Mexico 222 17.7 247 19.7 231 18.4 207 16.5 235 18.7 209 16.6 147 11.7 168 134 160 12.7 78 «6.2 
Ohio 25 54 121 26.1 93 20.1 84 18.1 96 20.7 103 22.2 67 14.5 7 1.5 1 0.2 63 13.6 


Rhode Island 130 18.1 162 22.6 148 20.6 116 16.2 87 12.1 92 12.8 78 10.9 67 9.3 6 08 64 8.9 
All States? 2,046 19.4 2,183 20.7 2,025 19.2 1,797 17.0 1,824 17.3 1,743 16.5 1,042 99 912 8.6 679 64 706 6.7 


’Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. °N-counts include individuals with missing data for state location who selected a grade level. 
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Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 N/A 

State n % n % n % n % n % n % n % n % n % n % 

Arkansas 628 22.1 578 204 526 185 428 15.1 373 131 413 146 414 146 392 13.8 169 6.0 158 5.6 
Colorado 36 «619.7 51 27.9 40 21.9 30 16.4 29 15.8 27 «14.8 34 18.6 31 169 31 169 11 60 
D.C. 4 13.3 8 26.7 9 30.0 3 10.0 2 6.7 4 13.3 3 10.0 3 =©10.0 1 3.3 5 16.7 
Illinois 388 21.5 387 21.5 375 20.8 342 19.0 360 20.0 335 18.6 64 3.5 14 0.8 60 3.3 83 4.6 
Maryland 318 30.1 326 30.9 327. 31.0 193 183 148 140 165 15.6 3 0.3 59 5.6 6 0.6 65 6.2 
Massachusetts 216 28.7 211 28.1 207 27.55 133 17.7. 115 15.3 120 16.0 46 6.1 -- -- 38 5.1 25 3.3 
Mississippi 6 8.3 8 11.1 5 6.9 14 19.4 9 12.5 16 22.2 1 14 20 27.8 1 1.4 7 9.7 
New Jersey 792, 21.7. +758 20.8 725 #199 615 169 534 146 543 149 282 7.7 252 6.9 259 7.1 182 5.0 
New Mexico 56 17.0 56 17.0 51 15.5 49 148 49 14.8 55 16.7 56 170 53 161 45 13.6 37 11.2 
Ohio 29 6.7 111 25.6 76 17.6 64 14.8 78 18.0 91 21.0 38 8.8 2 0.5 -- -- 75 17.3 
Rhode Island 9 20.5 15 34.1 8 18.2 3 6.8 4 9.1 7 15.9 1 2.3 1 2.3 -- -- 4 9.1 
All States® 2,496 22.2 2,519 22.4 2,359 21.0 1,882 16.8 1,709 15.2 1,784 159 943 8.4 827 7.4 612 54 654 58 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 


’N-counts include individuals with missing data for state location who selected a grade level. 
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Q4. Please indicate which assessment(s) you administered for Mathematics. Select all that apply.? 


Grade3 Grade4 Grade5 Grade6 Grade7 Grade8 ALG 1 GEO ALG 2 INT 1 INT 2 INT3 N/A 

State n % n % n % n % n % n % n % n % n % n % n % n % n % 

Arkansas 432 183 420 17.8 394 16.7 391 166 401 17.0 354 15.0 353 15.0 278 118 85 3.6 3 O21 3 O21 2 0.1 203 86 
Colorado 25 18.4 26 19.1 28 20.6 23 169 23 16.9 18 13.2 26 19.1 29 213 27 19.9 - - - - - - 11 «81 
D.C. 19 34.5 17 30.9 11 20.0 6 10.9 4 7.3 4 73 5 9.1 2 3.6 - - - - - - - - 2 3.6 
Illinois 497 22.0 477 21.1 473 209 446 19.7 467 206 439 19.4 95 4.2 7 03 84 3.7 130.6 3 O21 3 0.1 158 7.0 
Maryland 116 209 158 285 133 24.0 72 #130 81 146 74 #133 £68 «12.3 1 0.2 37 6.7 1 02 - - - - 49 88 
Massachusetts 174 25.1 179 258 169 244 164 23.7 141 203 134 19.3 40 5.8 15 2.22 17 25 2 03 - - - - 40 58 
Mississippi 18 19.6 16 17.4 8 8.7 12 130 10 109 9 98 14 15.2 - - 1 11 - - 1 11 1 1.1 24 26.1 
New Jersey 367 19.1 335 174 328 17.1 241 12.6 227 11.8 169 8.8 272 142 202 105 185 9.6 7 O04 5 03 4 0.2 171 89 
New Mexico 223 17.8 250 19.9 225 17.9 205 16.3 224 17.8 206 164 196 156 184 146 170 13.5 13 10 12 #10 12 1.0 101 8.0 
Ohio 100 216 121 26.1 95 20.5 84 181 85 184 87 188 53 114 32 6.9 2 O04 14 «93.0 1 02 - - 49 10.6 
Rhode Island 133 185 157 219 145 202 111 155 87 121 70 97 83 116 75 104 1 O12 - - - - - - 62 8.6 
All States” 2,109 20.0 2,164 20.5 2,014 19.1 1,758 16.7 1,756 16.7 1,568 1491,206 11.4 826 78 609 58 53 O5 25 O2 22 0.2 871 83 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
’N-counts include individuals with missing data for state location who selected a grade level. 
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Grade3 Grade4 Grade5 Grade6 Grade7 Grades ALG 1 GEO ALG 2 INT 1 INT 2 INT 3 N/A 
State n % n % n % n % n % n % n % n % n % n % n % n % n % 
Arkansas 624 22.0 582 20.5 528 18.6 422 14.9 354 12.5 366 12.9 453 16.0 416 14.7 128 45 1 00 1 00 1 O00 183 64 
Colorado 38 20.8 53 29.0 39 21.3 31 169 27 148 25 13.7 22 120 19 104 20 109 14 7.7 10 55 11 60 13 7.1 
D.C. 4 13.3 7 23.3 9 300 3 100 2 67 4 133 7 233 #- = 1 $33 - = -- -- so 1 3.3 
Illinois 390 21.6 388 21.5 379 21.0 348 19.3 356 19.7 341 18.9 45 2.5 9 05 49 2.7 20 11 #6 03 4 O2 85 4.7 
Maryland 324 30.7 327 31.0 323 30.6 192 18.2 140 13.3 142 13.4 115 109 - - 57 54 - = 1 O14 - - 59 56 
Massachusetts 224 29.8 209 27.8 215 28.6 132 17.6 116 154 114 152 51 68 23 31 17 23 2 03 - -- 1 O21 31 41 
Mississippi 5 69 8 11.1 7 97 #14 194 9 125 16 22.22 15 208 -- - 2 #28 - = 1 $14 -  - 7 9.7 


New Jersey 804 22.0 766 21.0 721 19.8 612 16.8 532 146 467 12.8 378 104 250 69 220 60 5 O12 5 O41 8 O02 160 44 
New Mexico 57 17.3 56 17.0 51 155 50 152 49 148 61 185 72 21.8 61 185 56 17.0 2 0.6 1 03 3 09 30 9.1 
Ohio 88 20.3 114 263 75 17.3 65 15.0 69 159 88 203 52 120 23 53 2 05 2 05 1 02 - - 38 88 
Rhode Island 10 22.7 15 34.1 8 182 3 68 4 91 6136 7 159 6 136 #14 23 -  - oo - - oo 
All States® 2,583 23.0 2,533 22.6 2,365 21.11,880 16.71,666 14.8 1,638 14.61,219 10.9 807 7.2 554 49 46 O04 26 O2 28 O2 609 54 
*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 

’N-counts include individuals with missing data for state location who selected a grade level. 
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Test Administrator’s Training Experience 


able G-6. TA Survey (PBA): Q1. Have you ever administered a computer-based test to students before (either 


lat the state or district level)? 


Yes No 

State n % n % 

Arkansas 845 35.9 1,506 64.1 
Colorado 97 71.3 39 28.7 
D.C. 26 47.3 29 52.7 
Illinois 1,268 56.1 991 43.9 
Maryland 380 68.5 175 31.5 
Massachusetts 219 31.7 471 68.3 
Mississippi 34 37.0 58 63.0 
New Jersey 503 26.3 1,406 73.7 
New Mexico 839 66.9 416 33.1 
Ohio 190 41.1 272 58.9 
Rhode Island 343 48.0 371 52.0 
All States 4,756 45.3 5,754 54.7 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-7. TA Survey (EOY): Q1. Have you ever administered a computer-based test to students before (either 


lat the state or district level)? 


Yes No 

State n % n % 

Arkansas 1,619 57.3 1,205 42.7 
Colorado 143 78.6 39 21.4 
D.C. 17 58.6 12 41.4 
Illinois 1,249 69.4 550 30.6 
Maryland 729 69.4 321 30.6 
Massachusetts 327 43.9 418 56.1 
Mississippi 40 55.6 32 44.4 
New Jersey 1,569 43.2 2,065 56.8 
New Mexico 258 78.4 71 21.6 
Ohio 246 56.9 186 43.1 
Rhode Island 36 83.7 7 16.3 
All States 6,255 55.9 4,926 44.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-8. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Accessibility Features & Accommodations Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 110 4.7 328 14.1 1,305 56.1 366 15.7 216 9.3 
Colorado 10 7.5 20 14.9 52 38.8 8 6.0 44 32.8 
D.C. -- -- 4 7.3 32 58.2 8 14.5 11 20.0 
Illinois 132 6.0 354 16.0 1,018 45.9 267 12.0 446 20.1 
Maryland 36 6.6 80 14.7 236 43.3 64 11.7 129 23.7 
Massachusetts 47 6.9 169 24.9 345 50.7 42 6.2 77 11.3 
Mississippi 6 6.7 12 13.5 40 44.9 16 18.0 15 16.9 
New Jersey 177 9.5 382 20.6 889 47.8 155 8.3 255 13.7 
New Mexico 124 10.0 289 23.4 576 46.7 76 6.2 169 13.7 
Ohio 49 10.7 107 23.3 209 45.5 21 4.6 73 15.9 
Rhode Island 65 9.4 131 18.8 311 44.7 56 8.1 132 19.0 
All States 758 73 1,884 18.3 5,026 48.7 1,083 10.5 1,572 15.2 


Administration of Computer-Based Assessments for Test Administrators Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 99 4.3 319 13.7 1,324 57.0 389 16.7 192 8.3 
Colorado 8 5.9 16 11.8 57 41.9 12 8.8 43 31.6 
D.C. 1 1.8 4 7.3 31 56.4 7 12.7 12 21.8 
Illinois 109 4.9 306 13.8 1,088 49.1 264 11.9 447 20.2 
Maryland 36 6.6 81 14.9 237 43.7 65 12.0 123 22.7 
Massachusetts 51 7.6 144 21.4 355 52.8 47 7.0 75 11.2 
Mississippi 6 6.8 12 13.6 44 50.0 15 17.0 11 12.5 
New Jersey 162 8.8 363 19.7 914 49.6 173 9.4 231 12.5 
New Mexico 117 9.5 277 22.5 604 49.1 85 6.9 148 12.0 
Ohio 46 10.0 110 24.0 221 48.1 27 5.9 55 12.0 
Rhode Island 56 8.1 130 18.8 339 49.1 59 8.6 106 15.4 
All States 693 6.7 1,770 17.2 5,228 50.8 1,147 11.2 1,447 14.1 
Infrastructure Trials: Running a Dress Rehearsal Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 121 5.2 281 12.1 1,134 48.7 536 23.0 257 11.0 
Colorado 10 75 20 14.9 28 20.9 12 9.0 64 47.8 
D.C. 2 3.6 2 3.6 23 41.8 12 21.8 16 29.1 
Illinois 129 5.9 283 12.8 788 35.7 319 14.5 686 31.1 
Maryland 40 73 81 14.9 196 36.0 65 11.9 163 29.9 
Massachusetts 64 9.5 150 22.3 235 34.9 70 10.4 155 23.0 
Mississippi 6 6.9 13 14.9 31 35.6 14 16.1 23 26.4 
New Jersey 170 9.2 309 16.7 779 42.1 246 13.3 346 18.7 
New Mexico 170 13.8 279 22.6 453 36.7 129 10.5 202 16.4 
Ohio 53 11.6 81 17.7 177 38.6 30 6.6 117 25.5 
Rhode Island 59 8.5 112 16.1 270 38.8 60 8.6 195 28.0 
All States 826 8.0 1,620 15.7 4,126 40.1 1,497 14.5 2,229 21.6 
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able G-8. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Introduction to PARCC Training Modules Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 93 4.0 304 13.1 1,328 57.2 369 15.9 229 9.9 
Colorado 5 3.7 21 15.6 47 34.8 12 8.9 50 37.0 
D.C. -- -- 2 3.7 29 53.7 7 13.0 16 29.6 
Illinois 104 4.7 275 12.5 1,062 48.3 273 12.4 485 22.1 
Maryland 33 6.1 67 12.5 244 45.4 49 9.1 144 26.8 
Massachusetts 53 8.0 142 21.4 327 49.2 47 7.1 96 14.4 
Mississippi 4 4.6 12 13.8 41 47.1 15 17.2 15 17.2 
New Jersey 153 8.3 336 18.2 919 49.9 162 8.8 273 14.8 
New Mexico 121 9.9 269 21.9 567 46.2 75 6.1 195 15.9 
Ohio 41 8.9 103 22.4 213 46.4 22 4.8 80 17.4 
Rhode Island 55 8.0 118 17.1 328 47.5 51 7.4 139 20.1 
All States 663 6.5 1,660 16.2 5,119 49.9 1,086 10.6 1,724 16.8 
PearsonAccess"™™ Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 102 4.4 301 13.0 1,268 54.7 356 15.3 29. 12.6 
Colorado 6 4.5 17 12.8 45 33.8 13 9.8 52 39.1 
D.C. 1 1.9 2 3.7 27 50.0 7 13.0 17 31.5 
Illinois 112 5.1 299 13.6 963 44.0 269 12.3 548 25.0 
Maryland 34 6.3 66 12.2 193 35.7 39 7.2 208 38.5 
Massachusetts 59 8.8 158 23.6 307 45.8 44 6.6 102 15.2 
Mississippi 5 5.7 13 14.8 43 48.9 11 12.5 16 18.2 
New Jersey 158 8.6 346 18.9 865 47.3 164 9.0 297 16.2 
New Mexico 118 9.6 287 23.4 506 41.2 70 5.7 247 20.1 
Ohio 44 9.6 96 21.1 210 46.1 23 5.0 83 18.2 
Rhode Island 57 8.4 113 16.7 307 45.4 55 8.1 144 21.3 
All States 698 6.8 1,705 16.7 4,745 46.4 1,055 10.3 2,013 19.7 
Personal Needs Profile (PNP) Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 122 5.3 323 14.0 1,010 43.7 238 10.3 617 26.7 
Colorado 6 4.5 18 13.4 28 20.9 7 5.2 75 56.0 
D.C. 1 1.8 3 5.5 22 40.0 6 10.9 23 41.8 
Illinois 119 5.4 298 13.6 625 28.5 138 6.3 1,013 46.2 
Maryland 39 7.2 73 13.5 144 26.7 23 4.3 261 48.3 
Massachusetts 63 9.5 140 21.1 183 27.5 24 3.6 255 38.3 
Mississippi 6 6.8 10 11.4 28 43.2 9 10.2 25 28.4 
New Jersey 171 9.4 350 19.2 576 31.7 83 4.6 641 35.1 
New Mexico 129 9.8 294 24.0 365 29.8 40 3.3 406 33.1 
Ohio 49 10.7 90 19.7 141 30.9 18 3.9 159 34.8 
Rhode Island 71 10.5 118 17.4 198 29.2 28 4.1 263 38.8 
All States 769 7.5 1,726 16.9 3,340 32.7 617 6.0 3,748 36.7 


(continued) 


Submitted September 28, 2015 Page G-8 


PARCC Quality of Test Administration 


able G-8. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Proctor Caching & TestNav Configuration Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 126 5.4 310 13.4 1,040 44.9 274 11.8 568 24.5 
Colorado 5 3.8 16 12.0 36 27.0 8 6.0 68 51.1 
D.C. - - 3 5.6 22 40.7 5 9.3 24 44.4 
Illinois 108 4.9 294 13.4 724 33.0 166 7.6 899 41.0 
Maryland 35 6.5 64 12.0 160 29.9 25 4.7 251 46.9 
Massachusetts 32 9.3 141 21.2 191 28.8 20 3.0 250 37.7 
Mississippi 5 5.7 12 13.6 35 39.8 11 12.5 25 28.4 
New Jersey 161 8.8 331 18.2 627 34.4 112 6.1 592 32.5 
New Mexico 129 10.5 281 22.9 408 33.3 46 3.7 363 29.6 
Ohio 47 10.4 89 19.6 151 33.3 11 2.4 156 34.4 
Rhode Island 63 9.3 115 17.0 217 32.0 39 5.8 244 36.0 
All States 742 7.3 1,663 16.3 3,621 35.3 717 7.0 3,453 33.9 
Student Readiness Resources for PARCC Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 122 5.3 340 14.7 1,119 48.4 298 12.9 435 18.8 
Colorado 5 3.7 18 13.3 39 28.9 11 8.1 62 45.9 
D.C. 1 1.8 1 1.8 25 445.5 5 9.1 23 41.8 
Illinois 121 5.5 310 14.1 837 38.2 183 8.3 741 33.8 
Maryland 41 7.6 73 13.5 174 32.3 35 6.5 216 40.1 
Massachusetts 56 8.4 139 20.9 247 37.1 32 4.8 192 28.8 
Mississippi 6 6.7 10 11.2 40 44.9 14 15.7 19 21.3 
New Jersey 173 9.5 343 18.8 728 39.9 120 6.6 459 25.2 
New Mexico 124 10.1 292 23.7 474 38.5 74 6.0 267 21.7 
Ohio 52 11.6 99 22.0 166 36.9 17 3.8 116 25.8 
Rhode Island 59 8.7 116 17.1 254 37.5 47 6.9 201 29.7 
All States 763 7.5 1,747 17.1 4,109 40.3 841 8.2 2,742 26.9 
Student Registration Import Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 125 5.4 285 12.4 961 41.7 231 10.0 701 30.4 
Colorado 5 3.8 17 12.8 24 18.0 7 5.3 80 60.2 
D.C. - - 4 7.3 20 36.4 5 9.1 26 47.3 
Illinois 108 4.9 264 12.1 584 26.7 132 6.0 1,097 50.2 
Maryland 37 6.9 65 12.1 121 22.5 18 3.3 297 55.2 
Massachusetts 56 8.4 133 20.0 176 26.4 20 3.0 281 42.2 
Mississippi 6 6.8 9 10.2 31 35.2 9 10.2 33 37.5 
New Jersey 151 8.3 313 17.2 585 32.1 82 4.5 689 37.9 
New Mexico 113 9.2 246 20.0 378 30.8 44 3.6 447 36.4 
Ohio 46 10.1 89 19.6 140 30.8 10 2.2 169 37.2 
Rhode Island 57 8.4 107 15.8 201 29.7 31 4.6 281 41.5 
All States 707 6.9 1,536 15.1 3,229 31.7 591 5.8 4,115 40.4 


(continued) 


Submitted September 28, 2015 Page G-9 


PARCC Quality of Test Administration 


able G-8. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Technology “Readiness” for Schools & Districts Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 141 6.1 300 13.0 974 42.2 276 12.0 616 26.7 
Colorado 5 3.7 13 9.7 31 23.1 6 4.5 79 59.0 
D.C. -- -- 3 5.5 21 38.2 5 9.1 26 47.3 
Illinois 122 5.6 270 12.4 627 28.7 157 7.2 1,009 46.2 
Maryland 47 8.8 74 13.9 128 24.0 28 5.3 256 48.0 
Massachusetts 65 9.8 130 19.5 178 26.7 24 3.6 269 40.4 
Mississippi 6 6.8 10 11.4 34 38.6 12 13.6 26 29.5 
New Jersey 176 9.6 328 18.0 593 32.5 122 6.7 608 33.3 
New Mexico 165 13.5 263 21.5 369 30.2 59 4.8 365 29.9 
Ohio 54 11.8 92 20.2 134 29.4 13 2.9 163 35.7 
Rhode Island 60 8.8 112 16.4 209 30.6 37 5.4 266 38.9 
All States 844 8.3 1,600 15.7 3,305 32.4 741 7.3. 3,697 36.3 
Training from my state department of education 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 242 10.5 445 19.2 803 34.7 169 7.3 654 28.3 
Colorado 10 7.5 20 14.9 19 14.2 10 7.5 75 56.0 
D.C. 3 5.6 4 7.4 18 33.3 5 9.3 24 44.4 
Illinois 286 13.1 390 17.8 415 18.9 86 3.9 1,014 46.3 
Maryland 59 11.0 87 16.2 94 17.5 15 2.8 282 52.5 
Massachusetts 128 19.1 156 23.3 137 20.4 10 1.5 239 35.7 
Mississippi 6 6.7 14 15.7 33 37.1 9 10.1 27 30.3 
New Jersey 378 20.5 420 22.8 393 21.3 55 3.0 595 32.3 
New Mexico 334 27.2 307 25.0 229 18.6 33 2.7 326 26.5 
Ohio 118 25.9 102 22.4 95 20.9 4 0.9 136 29.9 
Rhode Island 123 17.9 149 21.7 167 24.3 30 4.4 218 31.7 
All States 1,694 16.6 2,100 20.5 2,408 23.5 427 4.2 3,601 35.2 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 97 4.2 253 10.9 1,177 50.6 584 25.1 215 9.2 
Colorado 9 6.8 17 12.8 36 27.1 16 12.0 55 41.4 
D.C. 2 3.6 5 9.1 24 43.6 6 10.9 18 32.7 
Illinois 127 5.7 298 13.5 891 40.3 538 24.3 359 16.2 
Maryland 47 8.7 83 15.3 175 32.3 47 8.7 190 35.1 
Massachusetts 58 8.6 143 21.1 264 39.0 68 10.0 144 21.3 
Mississippi 4 4.5 7 8.0 47 53.4 19 21.6 11 12.5 
New Jersey 165 8.8 329 17.6 851 45.6 378 20.3 142 7.6 
New Mexico 140 11.3 290 23.5 506 41.0 159 12.9 139 11.3 
Ohio 42 9.2 77 16.8 200 43.7 88 19.2 51 11.1 
Rhode Island 87 12.5 123 17.6 276 39.6 107 15.4 104 14.9 
All States 783 7.6 1,630 15.8 4,457 43.2 2,017 19.5 1,443 13.9 


(continued) 
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able G-8. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Training from my school 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 88 3.8 180 7.7 1,249 56.7 756 32.5 51 2.2 
Colorado 9 6.6 19 14.0 68 50.0 35 25.7 5 3.7 
D.C. -- -- 5 9.1 34 61.8 14 25.5 2 3.6 
Illinois 95 4.3 203 9.1 976 44.0 825 37.2 121 5.5 
Maryland 32 5.9 56 10.2 288 52.7 139 25.4 32 5.9 
Massachusetts 34 5.0 118 17.3 350 51.4 152 22.3 27 4.0 
Mississippi 5 5.6 7 7.9 45 50.6 30 33.7 2 2.2 
New Jersey 121 6.5 258 13.9 934 50.2 507 27.3 40 2.2 
New Mexico 78 6.3 203 16.4 605 48.9 302 24.4 49 4.0 
Ohio 39 8.5 65 14.1 213 46.3 116 25.2 27 5.9 
Rhode Island 55 7.9 101 14.5 329 47.3 172 24.7 38 5.5 
All States 557 5.4 1,220 11.8 5,107 49.4 3,058 29.6 394 3.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-9. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Accessibility Features & Accommodations Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 124 4.4 387 13.8 1,647 58.7 364 13.0 284 10.1 
Colorado 8 4.5 31 17.3 91 50.8 14 7.8 35 19.6 
D.C. 1 3.4 1 3.4 21 82.8 -- -- 3 10.3 
Illinois 81 4.6 262 14.9 872 49.7 186 10.6 355 20.2 
Maryland 68 6.6 126 12.2 486 47.2 129 12.5 221 21.5 
Massachusetts 44 6.0 149 20.2 374 50.8 47 6.4 122 16.6 
Mississippi 3 4.2 10 13.9 42 58.3 8 11.1 9 12.5 
New Jersey 256 7.2 658 18.5 1,840 51.8 319 9.0 479 13.5 
New Mexico 24 7.5 64 20.2 160 49.8 17 5.3 55 17.1 
Ohio 44 10.5 107 25.4 179 42.5 19 4.5 72 17.1 
Rhode Island 2 4.7 4 9.3 28 65.1 4 9.3 5 11.6 
All States 656 6.0 1,806 16.4 5,769 52.5 1,110 10.1 1,642 15.0 


Administration of Computer-Based Assessments for Test Administrators Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 115 4.1 343 12.3 1,693 60.6 376 13.5 265 9.5 
Colorado 8 4.5 32 17.9 90 50.3 14 7.8 35 19.6 
D.C. -- -- 2 6.9 22 75.9 1 3.4 4 13.8 
Illinois 70 4.0 231 13.1 898 51.1 202 11.5 358 20.4 
Maryland 62 6.1 101 9.9 492 48.1 128 12.5 239 23.4 
Massachusetts 44 6.0 159 21.6 375 51.0 48 6.5 109 14.8 
Mississippi 3 4.2 8 11.1 49 68.1 5 6.9 7 9.7 
New Jersey 237 6.7 608 17.2 1,914 54.0 348 9.8 436 12.3 
New Mexico 23 7.2 57 17.9 172 54.1 20 6.3 46 14.5 
Ohio 43 10.3 95 22.7 188 45.0 26 6.2 66 15.8 
Rhode Island 2 4.7 4 9.3 28 65.1 6 14.0 3 7.0 
All States 608 5.6 1,648 15.1 5,947 54.3 1,177 10.8 1,568 14.3 
Infrastructure Trials: Running a Dress Rehearsal Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 137 4.9 339 12.1 1,416 50.6 536 19.2 369 13.2 
Colorado 10 5.6 36 20.2 62 34.8 15 8.4 55 30.9 
D.C. 1 3.4 2 6.9 17 58.6 2 6.9 7 24.1 
Illinois 88 5.0 239 13.6 645 36.8 215 12.3 564 32.2 
Maryland 77 7.5 136 13.2 376 36.6 130 12.7 308 30.0 
Massachusetts 41 5.6 141 19.2 290 39.5 68 9.3 194 26.4 
Mississippi 1 1.4 12 16.7 38 52.8 5 6.9 16 22.2 
New Jersey 276 7.8 618 17.4 1,529 43.1 470 13.2 655 18.5 
New Mexico 34 10.7 60 18.8 146 45.8 28 8.8 51 16.0 
Ohio 44 10.5 89 21.2 149 35.6 37 8.8 100 23.9 
Rhode Island 2 4.7 5 11.6 18 41.9 7 16.3 11 25.6 
All States 712 6.5 1,680 15.3 4,710 43.0 1,518 13.9 2,335 21.3 
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PARCC Quality of Test Administration 


able G-9. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Introduction to PARCC Training Modules Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 93 4.0 304 13.1 1,328 57.2 369 15.9 229 9.9 
Colorado 5 3.7 21 15.6 47 34.8 12 8.9 50 37.0 
D.C. -- -- 2 3.7 29 53.7 7 13.0 16 29.6 
Illinois 104 4.7 275 12.5 1,062 48.3 273 12.4 485 22.1 
Maryland 33 6.1 67 12.5 244 45.4 49 9.1 144 26.8 
Massachusetts 53 8.0 142 21.4 327 49.2 47 7.1 96 14.4 
Mississippi 4 4.6 12 13.8 41 47.1 15 17.2 15 17.2 
New Jersey 153 8.3 336 18.2 919 49.9 162 8.8 273 14.8 
New Mexico 121 9.9 269 21.9 567 46.2 75 6.1 195 15.9 
Ohio 41 8.9 103 22.4 213 46.4 22 4.8 80 17.4 
Rhode Island 55 8.0 118 17.1 328 47.5 51 7.4 139 20.1 
All States 596 5.5 1,640 15.1 5,747 52.8 1,094 10.1 1,801 16.6 
PearsonAccess"™™ Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 120 4.3 383 13.8 1566 56.4 359 12.9 350 12.6 
Colorado 9 5.1 35 19.8 81 45.8 13 7.3 39 22.0 
D.C. 1 3.4 1 3.4 17 58.6 1 3.4 9 31.0 
Illinois 74 4.2 227 13.0 823 47.1 169 9.7 456 26.1 
Maryland 57 5.6 116 11.4 365 35.9 94 9.2 386 37.9 
Massachusetts 43 5.9 148 20.5 341 47.2 78 6.6 143 19.8 
Mississippi 3 4.2 11 15.5 40 56.3 8 11.3 9 12.7 
New Jersey 252 7.2 636 18.1 1,794 51.2 295 8.4 528 15.1 
New Mexico 30 9.6 63 20.1 130 41.5 15 4.8 75 24.0 
Ohio 41 10.0 104 25.3 163 39.7 17 4.1 86 20.9 
Rhode Island 2 4.7 3 7.0 31 72.1 4 9.3 3 7.0 
All States 634 5.8 1,735 16.0 5,373 49.5 1,029 9.5 2,085 19.2 
Personal Needs Profile (PNP) Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 132 4.7 442 15.9 1,222 44.0 243 8.7 741 26.7 
Colorado 11 6.1 38 21.2 55 30.7 9 5.0 66 36.9 
D.C. 1 3.4 3 10.3 13 44.8 -- = 12 41.4 
Illinois 80 4.6 246 14.2 522 30.1 88 5.1 797 46.0 
Maryland 70 6.8 141 13.8 283 27.7 63 6.2 466 45.6 
Massachusetts 58 8.0 142 19.7 192 26.6 27 3.7 302 41.9 
Mississippi 1 1.4 11 15.5 35 49.3 8 11.3 16 22.5 
New Jersey 288 8.2 652 18.6 1,254 35.8 178 5.1 1,132 32.3 
New Mexico 27 8.6 64 20.3 109 34.6 10 3.2 105 33.3 
Ohio 52 12.7 94 22.9 107 26.0 12 2.9 146 35.5 
Rhode Island 1 2.4 6 14.3 17 40.5 3 7.1 15 35.7 
All States 725 6.7 1,846 17.0 3,828 35.3 643 5.9 3,804 35.1 


(continued) 
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able G-9. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Proctor Caching & TestNav Configuration Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 136 4.9 400 14.4 1,240 44.7 265 9.5 736 26.5 
Colorado 15 8.4 34 19.0 63 35.2 8 4.5 59 33.0 
D.C. 1 3.4 2 6.9 14 48.3 - = 12 41.4 
Illinois 83 4.8 228 13.1 602 34.5 101 5.8 729 41.8 
Maryland 73 7.1 105 10.3 295 28.9 73 7.1 475 46.5 
Massachusetts 61 8.5 128 17.9 206 28.8 29 4.1 291 40.7 
Mississippi 3 4.2 9 12.5 40 55.6 5 6.9 15 20.8 
New Jersey 293 8.4 605 17.3 1,314 37.5 215 6.1 1,076 30.7 
New Mexico 27 8.5 69 21.8 118 37.2 12 3.8 91 28.7 
Ohio 47 11.4 91 22.1 97 23.5 16 3.9 161 39.1 
Rhode Island 3 7.1 6 14.3 17 40.5 2 4.8 14 33.3 
All States 743 6.8 1,685 15.5 4,027 37.1 728 6.7 3,665 33.8 
Student Readiness Resources for PARCC Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 131 4.7 392 14.2 1,361 49.2 301 10.9 583 21.1 
Colorado 15 8.5 34 19.3 62 35.2 13 7.4 52 29.5 
D.C. 1 3.4 = = 17 58.6 == = 11 37.9 
Illinois 81 4.6 248 14.2 660 37.9 135 7.7 619 35.5 
Maryland 69 6.8 126 12.4 333 32.6 84 8.2 408 40.0 
Massachusetts 50 6.9 153 21.1 255 35.2 35 4.8 231 31.9 
Mississippi 3 4.2 13 18.1 35 48.6 6 8.3 15 20.8 
New Jersey 293 8.4 632 18.0 1,516 43.3 256 7.3 807 23.0 
New Mexico 29 9.2 59 18.7 135 42.7 17 5.4 76 24.1 
Ohio 49 11.9 92 22.3 118 28.6 23 5.6 131 31.7 
Rhode Island 2 4.7 7 16.3 20 46.5 3 7.0 11 25.6 
All States 724 6.7 1,764 16.3 4,533 41.8 877 8.1 2,949 27.2 
Student Registration Import Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 122 4.4 371 13.4 1,188 42.8 233 8.4 861 31.0 
Colorado 13 7.3 35 19.6 50 27.9 10 5.6 71 39.7 
D.C. 1 3.4 1 3.4 12 41.4 1 3.4 14 48.3 
Illinois 71 7.1 218 12.5 493 28.3 90 5.2 867 49.9 
Maryland 63 6.2 116 11.4 242 23.7 61 6.0 540 52.8 
Massachusetts 56 7.7 112 15.4 195 26.9 25 3.4 337 46.5 
Mississippi 2 2.8 12 16.7 36 50.0 4 5.6 18 25.0 
New Jersey 262 7.5 572 16.4 1,201 34.3 185 5.3 1,277 36.5 
New Mexico 24 7.6 60 19.0 115 36.4 14 4.4 103 32.6 
Ohio 44 10.7 79 19.1 99 24.0 13 3.1 178 43.1 
Rhode Island 2 4.8 5 11.9 17 40.5 2 4.8 16 38.1 
All States 661 6.1 1,589 14.7 3,665 33.8 641 5.9 4,289 39.5 


(continued) 
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able G-9. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Technology “Readiness” for Schools & Districts Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 143 5.2 406 14.6 1,189 42.9 293 10.6 741 26.8 
Colorado 16 9.0 40 22.5 47 26.4 5 2.8 70 39.3 
D.C. 1 3.4 - = 14 48.3 = = 14 48.3 
Illinois 84 4.9 232 13.4 498 28.8 109 6.3 805 46.6 
Maryland 77 7.5 140 13.7 255 24.9 68 6.6 485 47.3 
Massachusetts 59 8.2 144 20.0 183 25.4 28 3.9 306 42.5 
Mississippi 3 4.2 11 15.3 38 52.8 4 5.6 16 22.2 
New Jersey 301 8.6 606 17.3 1,294 37.0 238 6.8 1,063 30.4 
New Mexico 31 9.8 65 20.6 107 34.0 14 4.4 98 31.1 
Ohio 48 11.6 79 19.0 95 22.9 15 3.6 178 42.9 
Rhode Island 1 2.4 6 14.6 18 43.9 3 7.3 13 31.7 
All States 766 7.1 1,736 16.0 3,753 34.6 780 7.2. 3,797 35.1 
Training from my state department of education 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 248 8.9 552 19.9 1,021 36.8 176 6.3 776 28.0 
Colorado 25 14.0 33 18.5 37 20.8 10 5.6 73 41.0 
D.C. -- -- 1 3.4 11 37.9 1 3.4 16 55.2 
Illinois 190 10.9 312 17.9 363 20.9 69 4.0 805 46.3 
Maryland 118 11.6 152 14.9 197 19.3 49 4.8 503 49.4 
Massachusetts 93 12.8 168 23.0 127 17.4 22 3.0 319 43.8 
Mississippi 4 5.6 17 23.9 28 39.4 5 7.0 17 23.9 
New Jersey 623 17.7 767 21.8 892 25.4 147 4.2 1,084 30.9 
New Mexico 64 20.3 67 21.3 87 27.6 11 3.5 86 27.3 
Ohio 106 25.4 84 20.1 69 16.5 12 2.9 146 35.0 
Rhode Island 6 14.3 11 26.2 12 28.6 1 2.4 12 28.6 
All States 1,479 13.6 2,178 20.0 2,858 26.3 505 4.6 3,844 35.4 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 122 4.4 282 10.1 1,471 52.8 610 21.9 303 10.9 
Colorado 17 9.4 28 15.6 66 36.7 18 10.0 51 28.3 
D.C. 1 3.4 -- = 12 41.4 2 6.9 14 48.3 
Illinois 96 5.5 228 13.0 749 42.6 414 23.5 273 15.5 
Maryland 87 8.5 157 15.3 320 31.2 124 12.1 338 32.9 
Massachusetts 54 7.4 133 18.1 299 40.7 84 11.4 164 22.3 
Mississippi 2 2.8 9 12.7 28 53.5 11 15.5 11 15.5 
New Jersey 244 6.8 509 14.2 1,776 49.7 754 21.1 290 8.1 
New Mexico 31 9.7 51 16.0 148 46.4 38 11.9 51 15.5 
Ohio 28 6.6 69 16.3 185 43.7 96 22.7 45 10.6 
Rhode Island - - 8 18.6 23 53.5 8 16.6 4 9.3 
All States 682 6.2 1,485 13.5 5,105 46.5 2,166 19.7 1,547 14.1 


(continued) 
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able G-9. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


raining material effectively prepared me to administer the PARCC assessments.” 
Training from my school 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 110 3.9 222 7.9 1,573 56.2 819 29.3 75 2.7 
Colorado 7 3.9 21 11.7 96 53.6 49 27.4 6 3.4 
D.C. -- -- 1 3.4 19 65.5 9 31.0 -- -- 
Illinois 79 4.5 157 8.9 854 48.4 575 32.6 98 5.6 
Maryland 49 4.8 85 8.3 530 51.6 300 29.2 64 6.2 
Massachusetts 42 5.7 110 14.9 406 55.2 143 19.4 35 4.8 
Mississippi 4 5.6 4 5.6 43 60.6 16 22.5 4 5.6 
New Jersey 177 5.0 387 10.8 1,882 52.7 1,037 29.0 89 2.5 
New Mexico 17 5.3 33 10.3 165 51.7 85 26.6 19 6.0 
Ohio 24 5.7 57 13.5 196 46.4 118 28.0 27 6.4 
Rhode Island -- -- 3 7.1 20 47.6 19 45.2 -- -- 
All States 509 4.6 1,085 9.9 5,807 52.8 3,178 28.9 419 3.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-10. TA Survey (PBA): Q3. The PARCC online training(s) prepared me to resolve basic problems 


related to technology (e.g., logging students in, exiting the test, etc.). 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 208 8.9 536 22.8 1,243 53.0 181 7.7 178 7.6 
Colorado 9 6.7 32 23.7 68 50.4 4 3.0 22 16.3 
D.C. 1 1.9 9 16.7 32 59.3 5 9.3 7 13.0 
Illinois 202 9.0 611 27.2 1,000 44.6 147 6.6 284 12.7 
Maryland 79 14.5 146 26.7 191 35.0 23 4.2 107 19.6 
Massachusetts 95 13.9 235 34.3 284 41.5 22 3.2 49 7.2 
Mississippi 10 11.0 22 24.2 40 44.0 105 5.5 135 7.1 
New Jersey 253 13.3 605 31.9 799 42.1 105 5.5 135 7.1 
New Mexico 225 18.1 375 30.2 469 37.8 44 3.5 129 10.4 
Ohio 77 16.8 152 33.2 174 38.0 11 2.4 44 9.6 
Rhode Island 77 10.9 196 27.8 317 44.9 29 4.1 87 12.3 
All States 1,243 11.9 2,929 28.1 4,630 44.4 582 5.6 1,051 10.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-11. TA Survey (EOY): Q3. The PARCC online training(s) prepared me to resolve basic problems 


related to technology (e.g., logging students in, exiting the test, etc.). 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 238 8.5 694 24.7 1,464 52.2 198 7A. 213 7.6 
Colorado 22 12.2 57 31.5 72 39.8 8 4.4 22 12.2 
D.C. 1 3.3 5 16.7 18 60.0 1 3.3 5 16.7 
Illinois 172 9.7 490 27.6 785 44.2 108 6.1 220 12.4 
Maryland 135 13.0 269 25.8 397 38.1 70 6.7 171 16.4 
Massachusetts 99 13.3 264 35.5 299 40.2 18 2.4 64 8.6 
Mississippi 4 5.7 20 28.6 35 50.0 6 8.6 5 7.1 
New Jersey 433 12.0 1,049 29.1 1,677 46.5 189 5.2 261 7.2 
New Mexico 40 12.2 112 34.3 125 38.2 14 4.3 36 11.0 
Ohio 81 19.1 151 35.5 143 33.6 10 2.4 40 9.4 
Rhode Island 2 4.7 12 27.9 24 55.8 3 7.0 2 4.7 
All States 1,232 11.1 3,133 28.2 5,060 45.6 630 5.7 1,039 9.4 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Test Administration Experience 


able G-12. TA Survey (PBA): Q1a. The policies and procedures within the Test Administrator Manual were easy 


‘or me to understand. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 81 3.5 240 10.4 1,640 70.8 355 15.3 
Colorado 2 1.5 19 14.5 94 71.8 16 12.2 
D.C. 2 3.8 6 11.3 36 67.9 9 17.0 
Illinois 76 3.5 243 11.1 1,559 71.0 317 14.4 
Maryland 23 4.3 67 12.6 357 67.0 86 16.1 
Massachusetts 30 4.5 89 13.3 466 69.8 83 12.4 
Mississippi 4 4.5 4 4.5 60 67.4 21 23.6 
New Jersey 107 5.9 293 16.0 1,226 67.1 202 11.1 
New Mexico 65 5.3 157 12.8 867 70.9 133 10.9 
Ohio 27 6.0 84 18.6 304 67.3 37 8.2 
Rhode Island 30 4.3 86 12.3 509 72.9 73 10.5 
All States 447 4.4 1,297 12.7 7,136 69.8 1,339 13.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-13. TA Survey (EOY): Q1a. The policies and procedures within the Test Administrator Manual were easy 


for me to understand. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 77 2.8 281 10.2 1,999 72.2 411 14.8 
Colorado 4 2.2 22 12.4 137 77.0 15 8.4 
D.C. -- -- 2 7.4 24 88.9 -- -- 
Illinois 41 2.3 168 9.6 1,281 73.4 255 14.6 
Maryland 41 4.0 98 9.6 668 65.7 209 20.6 
Massachusetts 24 3.3 91 12.5 540 74.3 72 9.9 
Mississippi = = 4 5.6 50 70.4 17 23.9 
New Jersey 150 4.3 431 12.3 2,498 71.2 430 12.3 
New Mexico 13 4.0 36 11.1 239 73.5 37 11.4 
Ohio 19 4.6 78 18.8 281 67.9 36 8.7 
Rhode Island -- -- 5 11.6 29 67.4 9 20.9 
All States 371 3.4 1,218 11.2 7,778 71.6 1,496 13.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-14. TA Survey (PBA): Qib. The instructions (including scripts for administering the assessment) within 


he Test Administrator Manual were easy for me to implement. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 87 3.8 230 10.0 1,575 68.5 407 17.7 
Colorado 4 3.1 19 14.5 89 67.9 19 14.5 
D.C. 2 3.8 3 5.7 36 67.9 12 22.6 
Illinois 82 3.8 185 8.5 1,499 68.7 416 19.1 
Maryland 23 4.3 65 12.2 335 63.1 108 20.3 
Massachusetts 20 3.0 70 10.5 468 70.0 111 16.6 
Mississippi 4 4.4 6 6.7 59 65.6 21 23.3 
New Jersey 92 5.1 243 13.4 1,224 67.4 258 14.2 
New Mexico 67 5.5 144 11.8 865 70.8 146 11.9 
Ohio 23 5.1 66 14.7 310 68.9 51 11.3 
Rhode Island 23 3.3 69 9.9 505 72.3 101 14.5 
All States 427 4.2 1,107 10.9 6,984 68.6 1,658 16.3 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-15. TA Survey (EOY): Qib. The instructions (including scripts for administering the assessment) within 


he Test Administrator Manual were easy for me to implement. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 77 2.8 284 10.3 1,943 70.3 459 16.6 
Colorado 3 1.7 21 11.8 135 75.8 19 10.7 
D.C. -- -- 4 14.3 23 82.1 1 3.6 
Illinois 40 2.3 140 8.1 1,228 70.7 329 18.9 
Maryland 43 4.3 99 9.8 631 32.5 236 23.4 
Massachusetts 20 2.8 62 8.6 539 74.3 104 14.3 
Mississippi - = 5 7.0 48 67.6 18 25.4 
New Jersey 127 3.6 348 10.0 2,486 71.3 527 15.1 
New Mexico 15 4.6 29 9.0 243 75.2 36 11.1 
Ohio 20 4.8 56 13.6 292 70.7 45 10.9 
Rhode Island -- -- 1 2.3 31 72.1 11 25.6 
All States 347 3.2 1,051 9.7 7,630 70.5 1,789 16.5 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-16. TA Survey (PBA): Q1c. Students appeared to understand the directions | read to them during test 


administration. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 68 3.0 192 8.4 1,638 71.3 400 17.4 
Colorado 3 2.3 19 14.6 91 70.0 17 13.1 
D.C. 3 5.7 8 15.1 34 64.2 8 15.1 
Illinois 86 3.9 244 11.2 1,485 68.2 363 16.7 
Maryland 23 4.3 79 14.9 340 64.0 89 16.8 
Massachusetts 30 4.5 70 10.5 469 70.6 95 14.3 
Mississippi 5 5.6 8 8.9 57 63.3 20 22.2 
New Jersey 90 5.0 184 10.2 1,279 70.6 258 14.2 
New Mexico 45 3.7 129 10.6 923 75.8 120 9.9 
Ohio 26 5.8 66 14.7 316 70.5 40 8.9 
Rhode Island 26 3.7 78 11.2 503 72.1 91 13.0 
All States 406 4.0 1,081 10.6 7,157 70.5 1,508 14.9 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-17. TA Survey (EOY): Qic. Students appeared to understand the directions | read to them during test 


administration. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 60 2:2 240 8.7 1,997 72.4 461 16.7 
Colorado 6 3.4 19 10.8 129 73.3 22 12.5 
D.C. 1 3.7 3 11.1 21 77.8 2 7.4 
Illinois 34 2.0 148 8.5 1,250 72.0 304 17.5 
Maryland 41 4.1 117 11.6 641 63.8 206 20.5 
Massachusetts 16 2.2 80 11.0 536 73.7 95 13.1 
Mississippi -- -- 4 5.6 48 67.6 19 26.8 
New Jersey 117 3.4 292 8.4 2,564 73.4 518 14.8 
New Mexico 14 4.3 28 8.7 243 75.2 38 11.8 
Ohio 24 5.8 40 9.7 313 75.6 37 8.9 
Rhode Island 1 2.3 3 7.0 29 67.4 10 23.3 
All States 315 2.9 975 9.0 7,804 72.2 1,716 15.9 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-18. TA Survey (PBA): Q1d. The instructions | read to the students covered all of the information 


necessary to take the test. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 91 3.9 335 14.5 1,524 66.1 356 15.4 
Colorado 5 3.8 33 25.0 83 62.9 11 8.3 
D.C. 2 3.8 10 19.2 34 65.4 6 11.5 
Illinois 98 4.5 368 16.9 1,389 63.6 328 15.0 
Maryland 28 5.3 132 24.8 301 56.5 72 13.5 
Massachusetts 35 5.2 163 24.4 394 59.0 76 11.4 
Mississippi 3 3.4 7 7.9 57 64.0 22 24.7 
New Jersey 121 6.7 371 20.4 1,102 60.6 224 12.3 
New Mexico 87 7.1 265 21.7 112 63.2 97 7.9 
Ohio 35 7.8 110 24.4 270 60.0 35 7.8 
Rhode Island 40 5.7 117 16.8 465 66.6 76 10.9 
All States 548 5.4 1,917 18.8 6,411 63.0 1,308 12.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-19. TA Survey (EOY): Q1d. The instructions | read to the students covered all of the information 


necessary to take the test. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 85 3.1 411 14.9 1,864 67.4 405 14.6 
Colorado 7 3.9 39 21.9 115 64.6 17 9.6 
D.C. 1 3.6 5 17.9 22 78.6 -- -- 
Illinois 47 2.7 257 14.9 1,157 66.9 269 15.5 
Maryland 53 5.2 176 17.4 595 58.7 190 18.7 
Massachusetts 25 3.4 134 18.4 491 67.4 78 10.7 
Mississippi Hl 1.4 9 12.7 42 59.2 19 26.8 
New Jersey 156 4.5 585 16.7 2,325 66.5 432 12.3 
New Mexico 19 5.9 52 16.2 218 67.9 32 10.0 
Ohio 27 6.5 94 22.7 263 63.5 30 7.2 
Rhode Island 2 4.8 4 9.5 29 69.0 7 16.7 
All States 424 3.9 1,770 16.3 7,150 66.0 1,484 13.7 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-20. TA Survey (PBA): Q1la-1d Summary Table 


State Mean Standard Deviation 
Arkansas 11.88 2.29 
Colorado 11.49 2.07 
D.C. 11.75 2.27 
Illinois 11.79 2.29 
Maryland 11.61 2.39 
Massachusetts 11.57 2.25 
Mississippi 12.10 2.89 
New Jersey 11.38 2.41 
New Mexico 11.35 2.22 
Ohio 11.10 2.24 
Rhode Island 11.61 2.17 
All States 11.62 2.31 


Note. a = .87. Scale scores were computed by taking the sum of items 1a-1d. 


able G-21. TA Survey (EOY): Qla-1d Summary Table 


State Mean Standard Deviation 
Arkansas 11.92 2.17 
Colorado 11.58 1.94 
D.C. 11.29 2.05 
Illinois 12.01 2.08 
Maryland 11.92 2.44 
Massachusetts 11.69 2.01 
Mississippi 12.69 2.07 
New Jersey 11.69 2.24 
New Mexico 11.54 2.29 
Ohio 11.24 2.17 
Rhode Island 12.35 1.91 
All States 11.81 2.20 


Note. a = .88. Scale scores were computed by taking the sum of items 1a-1d. 
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Technology- 
Clarification on related Navigate Mark Answers & Exiting the Test After 
Instructions Problems Find/Use Tools throughTest Enter Responses When Finished Completed Test Other 
State n % n % n % n % n % n % n % n % 
Arkansas 388 16.5 1,212 51.4 618 26.2 360 15.3 494 20.9 1,224 51.9 800 33.9 105 4.5 
Colorado 32 23.5 84 61.8 53 39.0 37 27.2 40 29.4 74 54.4 44 32.4 14 10.3 
D.C. 16 29.1 21 38.2 24 43.6 9 16.4 19 34.5 19 34.5 11 20.0 3 5.5 
Illinois 576 25.4 1,330 58.7 799 35.3 500 22.1 710 31.4 1,232 54.4 640 28.3 148 6.5 
Maryland 139 25.0 370 66.7 247 44.5 144 25.9 211 38.0 255 45.9 211 38.0 59 10.6 
Massachusetts 189 27.3 477 68.8 299 43.1 171 24.7 237 34.2 389 56.1 180 26.0 56 8.1 
Mississippi 13 14.1 46 50.0 29 31.5 12 13.0 16 17.4 40 43.5 24 26.1 3 3.3 
New Jersey 408 21.3 1,129 58.8 662 34.5 408 21.3 505 26.3 999 52.0 710 37.0 149 7.8 
New Mexico 262 20.9 796 63.4 443 35.3 260 20.7 338 26.9 614 48.9 364 29.0 116 9.2 
Ohio 117 25.3 286 61.8 215 46.4 126 27.2 175 37.8 263 56.8 164 35.4 31 6.7 
Rhode Island 127 17.7 433 60.3 260 36.2 120 16.7 190 26.5 401 55.8 177 24.7 40 5.6 
All States” 2,275 21.6 6,207 58.9 3,665 34.8 2,156 20.4 2,942 27.9 5,528 52.4 3,339 31.7 725 6.9 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
>N-counts include individuals with missing data for state location who selected a response. 
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Table G-23. TA Survey (EOY): Q2. Please indicate if student(s) asked questions about the following topics. Select all that apply.* 


Technology- 
Clarification on related Navigate Mark Answers & Exiting the Test After 

Instructions Problems Find/Use Tools throughTest Enter Responses When Finished Completed Test Other 
State n % n % n % n % n % n % n % n % 
Arkansas 413 14.6 1,518 53.5 854 30.1 477 16.8 711 25.1 1,406 49.5 1,009 35.6 118 4.2 
Colorado 40 21.9 106 57.9 59 32.2 38 20.8 54 29.5 101 55.2 58 31.7 15 8.2 
D.C. 10 33.3 22 73.3 17 56.7 9 30.0 15 50.0 14 46.7 7 23.3 -- -- 
Illinois 333 18.5 1,031 57.2 624 34.6 373 20.7 609 33.8 955 53.0 478 26.5 135 7.5 
Maryland 227 21.5 686 65.0 415 39.3 216 20.5 386 36.6 457 43.3 386 36.6 94 8.9 
Massachusetts 186 24.7 519 69.0 327 43.5 185 24.6 294 39.1 426 56.6 198 26.3 68 9.0 
Mississippi 5 6.9 32 44.4 20 27.8 7 9.7 13 18.1 35 48.6 24 33.3 1 1.4 
New Jersey 641 17.6 2,160 59.2 1,207. 33.1 663 18.2 1,050 28.8 1,935 53.1 1,244 34.1 248 6.8 
New Mexico 73 22.1 189 57.3 105 31.8 69 20.9 99 30.0 166 50.3 116 35.2 26 7.9 
Ohio 90 20.8 273 63.0 168 38.8 95 21.9 138 31.9 236 54.5 145 33.5 38 8.8 
Rhode Island 8 18.2 25 56.8 16 36.4 6 13.6 16 36.4 24 54.5 8 18.2 5 11.4 
All States” 2,034 18.1 6,588 58.7 3,826 34.1 2,942 27.9 3,399 30.3 5,777 51.4 3,692 32.9 751 6.7 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
>N-counts include individuals with missing data for state location who selected a response. 
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Table G-24. TA Survey (PBA): Q3. Please indicate if any of the following technology-related problems occurred during testing. Be sure to respond based on 


the type of assessment you most recently administered (i.e., either PBA or EOY). Select all that apply.? 


System 
Device(s) Devices Disconnect or TestNav Accessibility 
Student Log- Stopped Worked Lost Internet Log-Out During Online Tools Features Did 
No Problems On Working Slowly Connectivity Administration Did Not Work Not Work Other 
State n % n % n % n % n % n % n % n % n % 
Arkansas 322 13.7 1,135 48.1 866 36.7 640 27.1 564 23.9 1,164 49.4 255 10.8 137 5.8 198 8.4 
Colorado 14 10.3 68 50.0 74 54.4 48 35.3 31 22.8 73 53.7 27 19.9 15 11.0 21 15.4 
D.C. 6 10.9 29 52.7. 21 38.2 12 21.8 9 16.4 17 30.9 6 10.9 4 7.3 5 9.1 
Illinois 207 9.1 1,195 52.8 997 440 626 27.7 +558 24.66 1,238 54.7 358 15.8 200 88 305 13.5 
Maryland 32 5.8 343 61.8 334 60.2 220 39.6 203 36.6 367 66.1 117 21.1 68 12.3. 81 14.6 
Massachusetts 55 7.9 398 57.4 369 56.2 236 34.1 187 27.0 389 56.1 158 22.8 79 11.4 105 15.2 
Mississippi 4 4.3 48 52.2 39 42.4 29 31.5 23 25.0 66 71.7 13 14.1 6 6.5 3 3.3 


New Jersey 222 11.6 942 49.1 741 38.6 535 27.9 407 21.2 891 46.4 347 18.1 158 8.2 257 13.4 
New Mexico 78 6.2 725 57.7 728 58.0 501 39.9 425 33.8 879 70.0 232 18.5 144 11.5 197 15.7 


Ohio 37 8.0 261 56.4 222 47.9 151 32.6 119 25.7 243 52.5 86 18.6 59 12.7. 78 16.8 
Rhode Island 84 11.7. 385 53.6 273 38.0 178 24.8 135 18.8 311 43.3 128 17.8 58 8.1 92 12.8 
All States? 1,065 10.1 5,548 52.6 4,675 443 3,188 30.2 2,676 25.4 5,652 53.6 1,734 164 933 8.8 1,343 12.7 


'Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
’N-counts include individuals with missing data for state location who selected a response. 
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Table G-25. TA Survey (EOY): Q3. Please indicate if any of the following technology-related problems occurred during testing. Be sure to respond based on 


the type of assessment you most recently administered (i.e., either PBA or EOY). Select all that apply.? 


System 
Device(s) Devices Disconnect or TestNav Accessibility 
Student Log- Stopped Worked Lost Internet Log-Out During Online Tools Features Did 
No Problems On Working Slowly Connectivity Administration Did Not Work — Not Work Other 
State n % n % n % n % n % n % n % n % n % 
Arkansas 334 11.8 1,379 48.6 1,133 39.9 971 34.22 760 26.8 1,437 50.6 361 12.77 164 58 212 7.5 

Colorado 12 6.6 111 60.7. 102 55.7 81 44.3 78 42.6 113 61.7 50 27.3 28 15.3 26 14.2 
D.C. 4 13.3 14 46.7 11 36.7. 12 40.0 12 40.0 14 46.7 7 23.3 1 3.3 4 13.3 
Illinois 219 12.1 933 51.7 738 409 514 285 442 24.55 958 53.1 264 14.6 147 8.2 181 10.0 
Maryland 67 6.3 659 62.4 573 54.3 1\418 39.6 372 35.2 696 65.9 154 14.6 78 7.4 117 11.1 
Massachusetts 47 6.3 453 60.2 385 51.2 288 38.3 286 38.0 427 56.8 139 18.5 83 11.0 90 12.0 
Mississippi 13 18.1 37 51.4 27 37.5 20 27.8 17 23.6 42 58.3 12 16.7 9 12.5 8 11.1 


New Jersey 537 14.7. 1,725 47.3 1,436 39.4 1,067 29.3 884 24.2 1,692 46.4 532 14.6 295 8.1 327 9.0 
New Mexico 35 10.6 186 56.4 165 50.0 139 42.1 110 33.3 209 63.3 61 18.5 41 12.4 38 11.5 


Ohio 42 9.7 232 53.6 203 46.9 146 33.7. 126 29.1 241 55.7 76 17.6 44 10.2 60 13.9 
Rhode Island 9 20.5 19 43.2 16 36.4 8 18.2 5 11.4 20 45.5 6 13.6 3 6.8 7 15.9 
All States? 1,322 11.8 5,772 51.4 4,807 42.8 3,680 32.8 3,106 27.7 5,875 52.3 1,668 149 896 8.0 1,073 9.6 


'Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
’N-counts include individuals with missing data for state location who selected a response. 
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Very Early On Time Rush to Finish Did Not Finish 
State n % n % n % n % 
Arkansas 1,297 56.8 946 41.4 35 1.5 7 0.3 
Colorado 66 50.8 56 43.1 3 2.3 5 3.8 
D.C. 19 37.3 28 54.9 4 7.8 -- -- 
Illinois 1,186 54.5 949 43.6 33 1.5 9 0.4 
Maryland 318 60.3 200 38.0 6 1.1 3 0.6 
Massachusetts 137 20.9 428 65.1 66 10.0 26 4.0 
Mississippi 25 29.4 57 67.1 3 3.5 -- -- 
New Jersey 901 50.4 844 47.3 30 1.7 11 0.6 
New Mexico 332 27.3 746 61.2 107 8.8 33 2.7 
Ohio 170 37.9 252 56.1 19 4.2 8 1.8 
Rhode Island 319 46.0 361 52.1 9 1.3 4 0.6 
All States 4,788 47.4 4,881 48.4 315 3.1 108 1.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-27. TA Survey (EOY): Q4. Did your students have sufficient time to finish the test? 


Very Early On Time Rush to Finish Did Not Finish 
State n % n % n % n % 
Arkansas 1,704 62.2 1,003 36.6 20 0.7 11 0.4 
Colorado 104 57.8 71 39.4 3 1.7 2 1.1 
D.C. 14 51.9 13 48.1 -- -- -- -- 
Illinois 971 56.3 731 42.4 21 1.2 1 0.1 
Maryland 661 65.4 338 33.5 11 1.1 -- -- 
Massachusetts 231 32.4 435 61.0 37 5.2 10 1.4 
Mississippi 23 32.4 45 63.4 2 2.8 1 1.4 
New Jersey 1,981 57.1 1,424 41.1 47 1.4 15 0.4 
New Mexico 89 27.8 202 63.1 23 7.2 6 1.9 
Ohio 159 38.7 239 58.2 13 3.2 -- -- 
Rhode Island 16 38.1 24 57.1 2 4.8 -- -- 
All States 5,972 55.6 4,546 42.3 180 1.7 46 0.4 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Accessibility Features and Accommodations 


able G-28. TA Survey (PBA): Q1. Did you read the “PARCC Accessibility Features and Accommodations Manual” 


prior to administration? 


Yes No 

State n % n % 

Arkansas 1,956 86.0 319 14.0 
Colorado 99 75.6 32 24.4 
D.C. 39 75.0 13 25.0 
Illinois 1,761 80.9 416 19.1 
Maryland 385 72.6 145 27.4 
Massachusetts 568 85.7 95 14.3 
Mississippi 65 75.6 21 24.4 
New Jersey 1,481 82.7 309 17.3 
New Mexico 1,010 82.9 209 17.1 
Ohio 377 83.8 73 16.2 
Rhode Island 588 85.5 100 14.5 
All States 8,355 82.8 1,740 17.2 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-29. TA Survey (EOY): Q1. Did you read the “PARCC Accessibility Features and Accommodations Manual” 


prior to administration? 


Yes No 

State n % n % 

Arkansas 2,266 82.8 471 17.2 
Colorado 141 78.3 39 21.7 
D.C. 24 85.7 4 14.3 
Illinois 1,379 79.6 354 20.4 
Maryland 730 72.5 277 27.5 
Massachusetts 582 81.4 133 18.6 
Mississippi 60 84.5 11 15.5 
New Jersey 2,860 82.5 607 17.5 
New Mexico 264 83.0 54 17.0 
Ohio 355 86.6 55 13.4 
Rhode Island 32 80.0 8 20.0 
All States 8,729 81.2 2,016 18.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-30. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


PARCC Accessibility Features & Accommodations Manual—3" Edition 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 58 2.6 206 9.3 1,457 65.6 210 9.5 290 13.1 
Colorado 2 1.6 10 8.1 58 46.8 8 6.5 46 37.1 
D.C. 1 2.0 2 4.1 32 65.3 5 10.2 9 18.4 
Illinois 61 2.9 221 10.5 1,156 55.2 160 7.6 498 23.8 
Maryland 16 3.1 49 9.6 260 50.7 46 9.0 142 27.7 
Massachusetts 24 3.8 102 16.2 365 58.1 31 4.9 106 16.9 
Mississippi 2 2.5 5 6.3 44 55.0 13 16.3 16 20.0 
New Jersey 98 5.8 288 17.0 907 53.5 94 5.5 309 18.2 
New Mexico 60 5.1 174 14.8 691 58.7 45 3.8 207 17.6 
Ohio 17 3.9 74 17.1 228 52.7 18 4.2 96 22.2 
Rhode Island 33 5.0 89 13.6 382 58.3 39 6.0 112 17.1 
All States 372 3.8 1,226 12.6 5,596 57.7 671 6.9 1,838 18.9 
Accessibility & Accommodations Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 53 2.4 235 10.7 1,315 59.6 185 8.4 417 18.9 
Colorado 2 1.6 15 12.2 43 35.0 2 1.6 61 49.6 
D.C. 1 2.0 1 2.0 28 57.1 4 8.2 15 30.6 
Illinois 73 3.5 242 11.6 924 44.3 117 5.6 728 34.9 
Maryland 20 4.0 53 10.5 195 38.5 27 5.3 211 41.7 
Massachusetts 31 5.0 120 19.3 303 48.8 16 2.6 151 24.3 
Mississippi 1 1.3 7 8.9 40 50.6 8 10.1 23 29.1 
New Jersey 110 6.6 286 17.1 781 46.7 76 4.5 418 25.0 
New Mexico 69 5.9 206 17.7 547 47.1 31 2.7 309 26.6 
Ohio 17 3.9 83 19.0 202 46.3 14 3.2 120 27.5 
Rhode Island 36 5.5 101 15.6 291 44.8 21 3.2 200 30.8 
All States 414 4.3 1,357 14.1 4,684 48.7 501 5.2 2,660 27.7 
Personal Needs Profile Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 61 2.8 258 11.8 1,070 49.0 150 6.9 646 29.6 
Colorado 2 1.6 10 8.1 32 26.0 3 2.4 76 61.8 
D.C. a 2.0 2 4.0 19 38.0 4 8.0 24 48.0 
Illinois 75 3.6 249 12.1 632 30.6 64 3.1 1,043 50.6 
Maryland 21 4.2 60 11.9 147 29.2 10 2.0 266 52.8 
Massachusetts 31 5.0 125 20.1 204 32.9 5 0.8 256 41.2 
Mississippi 1 1.3 5 6.6 34 44.7 5 6.6 31 40.8 
New Jersey 113 6.8 281 16.9 592 35.5 44 2.6 636 38.2 
New Mexico 66 5.7 222 19.2 405 35.0 24 2.1 440 38.0 
Ohio 17 3.9 79 18.3 151 35.0 14 3.2 171 39.6 
Rhode Island 37 5.8 102 15.9 215 33.6 15 2.3 271 42.3 
All States 426 4.5 1,401 14.7 3,514 36.8 338 3.5 3,868 40.5 


(continued) 
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able G-30. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


Personal Needs Profile Field Definitions 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 32 2.8 244 11.2 1,083 49.6 158 7.2 638 29.2 
Colorado 2 1.6 8 6.6 33 27.0 3 2.5 76 62.3 
D.C. 1 2.0 2 4.0 19 38.0 5 10.0 23 46.0 
Illinois 76 3.7 236 11.5 655 32.0 72 3.5 1,009 49.3 
Maryland 20 4.0 62 12.6 150 30.4 10 2.0 252 51.0 
Massachusetts 31 5.0 129 20.8 204 33.0 5 0.8 250 40.4 
Mississippi 1 1.3 6 7.7 36 46.2 4 5.1 31 39.7 
New Jersey 111 6.7 286 17.2 580 34.9 53 3.2 630 38.0 
New Mexico 68 5.9 213 18.5 419 36.4 28 2.4 422 36.7 
Ohio 18 4.2 81 18.8 157 36.5 12 2.8 162 37.7 
Rhode Island 37 5.8 103 16.1 235 36.7 16 25 249 38.9 
All States 428 4.5 1,378 14.5 3,584 37.7 366 3.9 3,750 39.4 
Training or resources from my state department of education 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 102 4.6 356 16.2 982 44.7 139 6.3 618 28.1 
Colorado 6 4.9 17 13.8 21 17.1 5 4.1 74 60.2 
D.C. 1 2.0 3 6.1 17 34.7 3 6.1 25 51.0 
Illinois 143 6.9 316 15.3 532 25.8 68 3.3 1,003 48.6 
Maryland 29 5.8 72 14.4 108 21.6 15 3.0 277 55.3 
Massachusetts 61 9.7 150 23.9 179 28.5 8 1.3 229 36.5 
Mississippi 2 2.6 12 15.6 30 39.0 7 9.1 26 33.8 
New Jersey 224 13.5 352 21.2 497 29.9 37 2.2 554 33.3 
New Mexico 199 17.1 295 25.4 311 26.7 21 1.8 337 29.0 
Ohio 54 12.4 89 20.5 139 32.0 4 0.9 148 34.1 
Rhode Island 80 12.4 118 18.3 197 30.5 23 3.6 227 35.2 
All States 906 9.5 1,790 18.7 3,018 31.5 332 3.5 3,527 36.8 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 60 2.7 193 8.7 1,246 56.3 426 19.3 287 13.0 
Colorado 8 6.6 15 12.3 34 27.9 10 8.2 55 45.1 
D.C. xf 2.0 4 8.0 22 44.0 4 8.0 19 38.0 
Illinois 92 4.4 228 10.9 876 41.8 404 19.3 497 23.7 
Maryland 31 6.1 68 13.3 163 31.9 31 6.1 218 42.7 
Massachusetts 41 6.5 123 19.5 250 39.7 51 8.1 165 26.2 
Mississippi 1 1.3 5 6.4 44 56.4 14 17.9 14 17.9 
New Jersey 137 8.0 267 15.6 837 49.0 251 14.7 216 12.6 
New Mexico 103 8.7 247 20.9 530 45.0 96 8.1 203 17.2 
Ohio 21 4.8 61 13.9 210 47.8 65 14.8 82 18.7 
Rhode Island 65 9.9 112 17.0 289 43.9 71 10.8 122 18.5 
All States 563 5.8 1,328 13.7 4,512 46.4 1,428 14.7 1,886 19.4 


(continued) 
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able G-30. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 
Training from my school 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 49 2.2 139 6.3 1,338 60.2 575 25.9 120 5.4 
Colorado 5 3.9 17 13.4 62 48.8 28 22.0 15 11.8 
D.C. 1 2.0 5 9.8 34 66.7 11 21.6 -- -- 
Illinois 69 3.3 165 7.8 978 46.5 629 29.9 264 12.5 
Maryland 31 6.0 42 8.2 267 51.9 96 18.7 78 15.2 
Massachusetts 34 5.4 82 13.0 350 55.3 117 18.5 50 7.9 
Mississippi 1 1.3 4 5.0 49 61.3 18 22.5 8 10.0 
New Jersey 108 6.3 221 13.0 918 53.8 338 19.8 120 7.0 
New Mexico 66 5.6 169 14.3 632 53.3 216 18.2 102 8.6 
Ohio 22 5.0 51 11.6 223 50.8 82 18.7 61 13.9 
Rhode Island 44 6.7 81 12.3 336 50.9 125 18.9 74 11.2 
All States 432 4.4 981 10.1 5,200 53.3 2,244 23.0 894 9.2 


Note. Percentages are based on the valid responses. N-counts for “All States” include individuals with missing data 
for state location who selected a response. 
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able G-31. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


PARCC Accessibility Features & Accommodations Manual—3" Edition 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 58 2.2 251 9.5 1,687 64.0 249 9.4 390 14.8 
Colorado 3 1.8 26 15.5 106 63.1 4 2.4 29 17.3 
D.C. - = 4 14.3 15 53.6 2 7.1 7 25.0 
Illinois 30 1.8 170 10.2 947 56.6 134 8.0 393 23.5 
Maryland 23 2.4 89 9.2 528 54.7 85 8.8 241 24.9 
Massachusetts 17 2.5 104 15.4 359 53.3 31 4.6 163 24.2 
Mississippi -- -- 6 8.6 48 68.6 6 8.6 10 14.3 
New Jersey 135 4.1 455 13.9 1,946 59.3 189 5.8 558 17.0 
New Mexico 19 6.1 38 12.3 177 57.3 14 4.5 61 19.7 
Ohio 20 5.1 85 21.5 204 51.6 19 4.8 67 17.0 
Rhode Island -- -- 5 12.5 23 57.5 3 7.5 9 22.5 
All States 308 3.0 1,237 12.0 6,063 59.0 741 7.2 1,931 18.8 
Accessibility & Accommodations Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 60 2.3 287 10.9 1,544 58.7 197 75 542 20.6 
Colorado 3 1.8 29 17.6 84 50.9 4 2.4 45 27.3 
D.C. -- -- 1 3.6 17 60.7 2 7.1 8 28.6 
Illinois 35 2.1 186 11.2 755 45.4 88 5.3 598 36.0 
Maryland 22 2.3 109 11.4 422 44.2 63 6.6 339 35.5 
Massachusetts 19 2.8 115 17.2 289 43.3 21 3.1 223 33.4 
Mississippi -- -- 8 11.4 44 62.9 4 5.7 14 20.0 
New Jersey 155 4.8 504 15.5 1,650 50.9 151 4.7 782 24.1 
New Mexico 20 6.5 41 13.3 153 49.7 7 2.3 87 28.2 
Ohio 23 5.9 87 22.3 153 39.1 15 3.8 113 28.9 
Rhode Island -- -- 6 15.0 21 52.5 -- -- 13 32.5 
All States 340 3.3 1,379 13.5 5,151 50.5 557 5.5 2,767 27.1 
Personal Needs Profile Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 65 2.5 332 12.7 1,262 48.5 151 5.8 794 30.5 
Colorado 4 2.4 28 16.8 61 36.5 4 2.4 70 41.9 
D.C. -- -- 3 10.7 9 32.1 ul 3.6 15 53.6 
Illinois 40 2.4 218 13.2 535 32.4 49 3.0 811 49.1 
Maryland 31 3.2 123 12.9 266 27.9 38 4.0 497 52.0 
Massachusetts 19 2.9 120 18.1 198 29.9 16 2.4 310 46.8 
Mississippi -- -- 7 10.0 41 58.6 5 7.1 17 24.3 
New Jersey 170 5.3 539 16.6 1,237 38.2 111 3.4 1,181 36.5 
New Mexico 19 6.2 44 14.4 121 39.5 10 3.3 112 36.6 
Ohio 24 6.3 86 22.4 118 30.7 9 2.6 147 38.3 
Rhode Island -- -- 8 20.5 13 33.3 -- -- 18 46.2 
All States 374 3.7 1,517 15.0 3,874 38.2 298 3.9 3,979 39.2 
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able G-31. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


Personal Needs Profile Field Definitions 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 62 2.4 324 12.5 1,269 48.9 154 5.9 788 30.3 
Colorado 4 2.4 27 16.2 58 34.7 4 2.4 74 44.3 
D.C. -- -- 3 11.5 9 34.6 1 3.8 13 50.0 
Illinois 44 2.7 215 13.1 528 32.3 47 2.9 801 49.0 
Maryland 30 3.1 119 12.5 281 29.5 43 4.5 481 50.4 
Massachusetts 17 2.6 114 17.1 205 30.8 16 2.4 314 47.1 
Mississippi == = 8 12.1 40 60.6 3 4.5 15 22.7 
New Jersey 169 5.3 539 16.8 1,217 37.9 107 3.3 1,176 36.7 
New Mexico 16 5.2 45 14.7 126 41.0 8 2.6 112 36.5 
Ohio 28 7.2 81 20.9 124 32.0 9 2.3 145 37.5 
Rhode Island -- -- 8 20.0 15 37.5 -- -- 17 42.5 
All States 372 3.7 1,491 14.8 3,886 38.5 396 3.9 3,943 39.1 
Training or resources from my state department of education 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 126 4.8 438 16.8 1,165 44.7 150 5.8 727 27.9 
Colorado 9 5.4 33 19.9 51 30.7 6 3.6 67 40.4 
D.C. -- -- 2 7.1 10 35.7 1 3.6 15 53.6 
Illinois 96 5.8 253 15.3 439 26.6 48 2.9 813 49.3 
Maryland 56 5.9 140 14.9 232 24.6 41 4.4 473 50.2 
Massachusetts 47 7.1 142 21.5 146 22.1 15 2.3 310 47.0 
Mississippi 3 4.3 11 15.9 33 47.8 6 8.7 16 23.2 
New Jersey 317 9.8 658 20.4 1,065 33.1 115 3.6 1,065 33.1 
New Mexico 37 12.2 63 20.8 94 31.0 11 3.6 98 32.3 
Ohio 53 13.7 94 24.4 91 23.6 7 1.8 141 36.5 
Rhode Island 2 5.0 10 25.0 12 30.0 1 25 15 37.5 
All States 747 7.4 1,856 18.4 3,351 33.2 403 4.0 3,747 37.1 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 71 2.7 251 9.5 1,485 56.1 461 17.4 378 14.3 
Colorado 8 4.7 26 15.4 66 39.1 14 8.3 55 32.5 
D.C. -- -- 3 11.1 8 29.6 2 7.4 14 51.9 
Illinois 62 3.7 186 11.1 725 43.4 294 17.6 402 24.1 
Maryland 51 5.4 137 14.4 323 34.0 82 8.6 358 37.6 
Massachusetts 34 5.0 112 16.5 270 39.9 57 8.4 204 30.1 
Mississippi 1 14 7 10.0 39 55.7 10 14.3 13 18.6 
New Jersey 169 5.1 461 13.9 1,671 50.4 575 17.3 441 13.3 
New Mexico 28 9.1 43 14.0 144 46.9 32 10.4 60 19.5 
Ohio 23 5.8 54 13.6 186 46.9 75 18.9 59 14.9 
Rhode Island 1 2.5 5 12.5 17 42.5 6 15.0 11 27.5 
All States 449 4.4 1,291 12.5 4,955 48.1 1,612 15.6 2,000 19.4 
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able G-31. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 
Training from my school 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 67 2.5 179 6.7 1,626 61.3 629 23.7 153 5.8 
Colorado 8 4.7 16 9.3 100 58.1 39 22.7 9 5.2 
D.C. -- -- 1 3.7 20 74.1 5 18.5 1 3.7 
Illinois 50 3.0 124 7.4 847 50.5 430 25.7 225 13.4 
Maryland 28 2.9 66 6.9 549 57.0 214 22.2 106 11.0 
Massachusetts 28 4.1 86 12.6 388 56.8 98 14.3 83 12.2 
Mississippi 2 2.9 5 7.2 43 62.3 14 20.3 5 7.2 
New Jersey 124 3.7 336 10.1 1,825 54.8 794 23.9 250 7.5 
New Mexico 19 6.1 23 7.4 169 54.2 67 21.5 34 10.9 
Ohio 19 4.8 46 11.6 193 48.7 85 21.5 53 13.4 
Rhode Island 1 2.6 2 5.1 15 38.5 13 33.3 8 20.5 
All States 347 3.4 888 8.6 5,797 56.0 2,395 23.1 931 9.0 


Note. Percentages are based on the valid responses. N-counts for “All States” include individuals with missing data 
for state location who selected a response. 
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able G-32. TA Survey (PBA): Q3. Did you administer the following accessibility features or accommodations to any of your students? Select all that apply.* 


Test Calculator on 
Human Directions Word Non- 
Human Reader Native Text-to- Extended Prediction Calculator Other Tech 
Reader Math English Human Scribe Human Signer Language Speech Time Device Section Device 

State n % n % n % n % n % n % n % n % n % n % 
Arkansas 144 6.1 74 3.1 13 0.6 8 0.3 100 4.2 145 6.1 630 26.7 9 0.4 211 8.9 27 1.1 
Colorado 6 4.4 3 2.2 5 3.7 al 0.7 6 44 10 7.4 34 25.0 5 3.7 6 4.4 2 1.5 
D.C. 8 14.5 4 7.3 3 5.5 1 1.8 3 5.5 2 3.6 22 40.0 -- -- 8 14.5 3 5.5 
Illinois 238 10.5 100 4.4 208 9.2 7 0.3. 129 5.7. 128 5.7. 769 340 18 0.8 330 146 36 1.6 
Maryland 71 12.8 61 11.0 65 11.7 2 0.4 18 3.2 25 45 204 368 1 0.2 105 18.9 17 3.1 
Massachusett 
S 80 11.5 57 8.2 61 8.8 3 0.4 34 49 35 5.1 276 39.8 6 0.9 69 10.0 6 0.9 
Mississippi 1 1.1 1 1.1 1 1.1 -- -- -- -- -- -- 130 «14.1 -- -- 1 1.1 -- -- 


New Jersey 79 4.1 70 3.6 50 2.6 3 0.2 82 4.3 103 5.4 623 32.4 17 0.9 367 19.1 34 1.8 
New Mexico 90 7.2 53 42 31 2.5 7 06 71 5.7. 102 8.1 334 26.6 4 0.3 108 8.6 28 2.2 
Ohio 19 4.1 10 2.2 43 9.3 3 0.6 22 48 52 11.2 165 35.6 3 06 81 17.5 6 1.3 
Rhode Island 37 5.2 16 2.2 14 1.9 -- -- 19 2.6 33 46 213 29.7 8 11 37 5.2 9 1.3 
All States” 775 7.3 451 4.3 495 47 35 0.3 484 4.6 636 6.0 3,288 31.2 71 0.7 1,327 12.6 168 1.6 
@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 

’N-counts include individuals with missing data for state location who selected a response. 
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able G-33. TA Survey (EOY): Q3. Did you administer the following accessibility features or accommodations to any of your students? Select all that apply. 


Test Calculator on 
Human Directions Word Non- 
Human Reader Native Text-to- Extended Prediction Calculator Other Tech 
Reader Math English Human Scribe Human Signer Language Speech Time Device Section Device 

State n % n % n % n % n % n % n % n % n % n % 
Arkansas 174 6.1 74 2.6 24 0.8 6 0.2 107 3.8 185 6.5 774 27.3 8 0.3 231 8.1 32 1.1 
Colorado 10 5.5 4 2.2 4 2.2 2 11 10 5.5 13 7.1 38 20.8 8 4.4 6 3.3 5 2.7 
D.C. 7 = 23.3 1 3.3 2 6.7 -- -- 1 3.3 3 10.0 18 60.0 2 6.7 10 33.3 3 10.0 
Illinois 180 10.0 86 4.8 112 6.2 3 0.2 70 3.9 113 6.3 562 31.2 16 0.9 292 162 25 14 
Maryland 179 17.0 142 13.4 162 15.3 3 0.3 28 2.7 44 4.2 432 40.9 8 0.8 281 266 15 1.4 
Massachusett 

S 80 10.6 44 5.9 51 6.8 -- -- 21 2.8 35 4.7 273 36.3 5 0.7 63 8.4 8 1.1 
Mississippi 2 2.8 2 2.8 1 1.4 HT 14 4 5.6 2 2.8 15 20.8 -- -- 2 2.8 1 14 


New Jersey 194 5.3 158 4.3 86 2.4 9 0.2 151 4.1 250 6.9 1,261 346 25 0.7 814 22.3 48 1.3 
New Mexico 39 11.8 21 6.4 9 2.7 2 06 28 8.5 23 7.0 101 30.6 3 09 37 11.2 8 2.4 
Ohio 35 8.1 19 44 32 7.4 1 0.2 24 55 56 12.9 164 37.9 1 0.2 73 16.9 5 1.2 
Rhode Island 2 4.5 -- -- 1 2.3 -- -- 1 2.3 1 2.3 13 (29.5 -- -- 2 4.5 -- -- 
All States” 906 8.1 553 4.9 486 4.3 27 0.2 449 4.0 730 6.5 3,664 326 76 0.7 1,820 16.2 151 1.3 
@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 

’N-counts include individuals with missing data for state location who selected a response. 
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able G-34. TA Survey (PBA): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Appendix A: Accessibility Features & Accommodations for Students taking the Paper-Based PARCC Assessments 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 37 1.7 68 3.2 535 25.0 71 3.3 1,432 66.8 
Colorado -- -- 4 3.4 19 16.0 -- -- 96 80.7 
D.C. 1 2.0 2 4.0 12 24.0 3 6.0 32 64.0 
Illinois 34 1.7 64 3.2 368 18.5 53 2.7 1,468 73.9 
Maryland 12 2.5 11 2.3 78 16.0 14 2.9 373 76.4 
Massachusetts 12 2.0 31 5.2 121 20.1 8 1.3 429 71.4 
Mississippi 2 2.5 3 3.7 22 27.2 3 3.7 51 63.0 
New Jersey 46 2.9 100 6.4 285 18.2 31 2.0 1,101 70.4 
New Mexico 31 2.8 54 4.9 228 20.6 16 1.4 776 70.2 
Ohio 10 2.4 23 5.5 80 19.3 5 1.2 297 71.6 
Rhode Island 14 2.2 29 4.7 122 19.6 14 2.2 444 71.3 
All States 199 2.2 392 4.3 1,877 20.4 218 2.4 6,518 70.8 


Appendix B: Test Administration Protocol for the Human Reader Accommodations for ELA/L and the Human 
Reader Accessibility Feature for Math 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 39 1.8 76 3.6 528 24.8 64 3.0 1,425 66.8 
Colorado -- -- 5 4.2 14 11.9 1 0.8 98 83.1 
D.C. 1 2.0 1 2.0 13 26.5 4 8.2 30 61.2 
Illinois 39 2.0 77 3.9 385 19.5 60 3.0 1,413 71.6 
Maryland 12 2.5 14 2.9 97 20.0 14 2.9 348 71.8 
Massachusetts 13 2.2 31 5.2 128 21.4 12 2.0 414 69.2 
Mississippi 1 1.3 3 3.8 22 27.5 1 1.3 53 66.3 
New Jersey 55 3.5 90 5.8 251 16.1 28 1.8 1,132 72.8 
New Mexico 34 3.1 59 5.4 213 19.4 14 1.3 779 70.9 
Ohio 13 3.2 23 5.6 83 20.3 4 0.9 285 69.9 
Rhode Island 13 2.1 33 5.3 104 16.7 14 2.3 458 73.6 
All States 220 2.4 414 4.5 1,843 20.1 217 2.4 6,455 70.6 
Appendix C: Protocol for the Use of the Scribe and for Transcribing Student Responses 
Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 
Arkansas 33 1.6 64 3.0 477 22.5 52 2.4 1,497 70.5 
Colorado -- -- 3 2.6 18 15.4 -- -- 96 82.1 
D.C. 1 2.0 - - 12 24.5 3 6.1 33 67.3 
Illinois 40 2.0 76 3.8 382 19.3 58 2.9 1,423 71.9 
Maryland 11 2.2 15 3.1 98 20.0 15 3.1 351 71.6 
Massachusetts 12 2.0 34 5.7 106 17.8 11 1.8 432 72.6 
Mississippi 1 1.3 3 3.8 18 22.8 1 1.3 56 70.9 
New Jersey 47 3.0 91 5.9 229 14.8 25 1.6 1,156 74.7 
New Mexico 33 3.0 55 5.0 182 16.6 14 1.3 811 74.1 
Ohio 12 2.9 27 6.6 94 23.0 7 1.7 269 65.8 
Rhode Island 13 2.1 25 4.0 100 16.2 11 1.8 470 75.9 
All States 203 2.2 396 4.3 1,722 18.9 198 2.2 6,613 72.4 


(continued) 
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able G-34. TA Survey (PBA): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Appendix E: Guidance for Selecting and Administering the Extended Time Accommodation 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 39 1.8 70 3.3 783 36.7 110 5.2 1,129 53.0 
Colorado 1 0.8 6 5.1 25 21.2 2 1.7 84 71.2 
D.C. 1 2.0 2 4.1 19 38.8 5 10.2 22 44.9 
Illinois 43 2.2 91 4.6 630 31.9 111 5.6 1,103 55.8 
Maryland 10 2.1 23 4.7 151 31.1 29 6.0 273 56.2 
Massachusetts 16 2.7 37 6.2 215 35.9 19 3.2 312 52.1 
Mississippi 1 1.3 3 3.8 22 27.5 6 7.5 48 60.0 
New Jersey 53 3.4 124 8.0 471 30.3 71 4.6 837 53.8 
New Mexico 36 3.2 73 6.6 332 30.0 28 2.5 639 57.7 
Ohio 12 2.9 27 6.6 159 38.6 13 3.2 201 48.8 
Rhode Island 16 2.6 27 4.3 202 32.2 26 4.1 356 56.8 
All States 228 2.5 486 5.3 3,016 32.9 421 4.6 5,022 54.7 
Appendix I: PARCC ELA Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 38 1.8 78 3.7 653 30.7 88 4.1 1,268 59.7 
Colorado 1 0.8 5 4.2 25 21.0 2 1.7 86 72.3 
D.C. 1 2.0 1 2.0 10 20.4 4 8.2 33 67.3 
Illinois 42 2.1 92 4.7 436 22.0 67 3.4 1,341 67.8 
Maryland 10 2.1 15 3.1 107 22.2 15 3.1 336 69.6 
Massachusetts 19 3.2 39 6.5 139 23.3 11 1.8 389 65.2 
Mississippi 1 1.2 5 6.2 24 29.6 3 3.7 48 59.3 
New Jersey 58 3.7 108 7.0 353 22.7 38 2.4 995 64.1 
New Mexico 37 3.4 79 7.2 277 25.1 20 1.8 689 62.5 
Ohio 13 3.2 38 9.3 116 28.4 7 1.7 234 57.4 
Rhode Island 18 2.9 32 5.1 142 22.7 18 2.9 415 66.4 
All States 238 2.6 495 5.4 2,287 25.0 274 3.0 5,854 64.0 
Appendix J: PARCC Math Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 40 1.9 74 3.5 699 33.0 97 4.6 1,208 57.0 
Colorado 2 1.7 6 5.1 29 24.6 1 0.8 80 67.8 
D.C. 1 2.1 1 2.1 9 18.8 6 12.5 31 64.6 
Illinois 42 2.1 93 4.7 461 23.4 68 3.5 1,306 66.3 
Maryland 12 2.5 18 3.7 105 21.5 17 3.5 336 68.9 
Massachusetts 18 3.0 41 6.9 152 25.5 11 1.8 373 62.7 
Mississippi 1 1.3 4 5.0 22 27.5 2 2.5 51 63.8 
New Jersey 54 3.5 110 7.1 358 23.1 39 2.5 989 63.8 
New Mexico 41 3.7 84 7.6 283 25.8 21 1.9 670 61.0 
Ohio 13 3.2 40 9.8 126 31.0 7 1.7 221 54.3 
Rhode Island 15 2.4 35 5.6 143 23.1 16 2.6 411 66.3 
All States 239 2.6 509 5.6 2,393 26.2 286 3.1 5,696 62.4 
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able G-34. TA Survey (PBA): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Assistive Technology Guidelines 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 40 1.9 79 3.7 604 28.6 68 3.2 1,323 62.6 
Colorado 2 1.7 9 7.6 16 13.4 - - 92 77.3 
D.C. 1 2.0 2 4.1 10 20.4 4 8.2 32 65.3 
Illinois 40 2.0 81 4.1 343 17.5 50 2.6 1,441 73.7 
Maryland 10 2.1 18 3.7 79 16.3 8 1.6 370 76.3 
Massachusetts 18 3.0 38 6.4 111 18.8 9 1.5 416 70.3 
Mississippi 1 1.3 3 3.8 22 27.5 1 1.3 53 66.3 
New Jersey 55 3.5 110 7.1 278 17.9 33 2.1 1,075 69.3 
New Mexico 42 3.9 74 6.8 211 19.4 17 1.6 746 68.4 
Ohio 11 2.7 37 9.1 87 21.4 5 1.2 266 65.5 
Rhode Island 15 2.4 28 4.6 114 18.6 15 2.4 442 72.0 
All States 235 2.6 482 5.3 1,883 20.7 210 2.3 6,275 69.1 


PARCC Technical Assistance Bulletin - PARCC Assessments and Students with Visual Impairment, Including 
Blindness 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 36 1.7 64 3.0 482 22.8 47 2.2 1,485 70.2 
Colorado -- -- 3 2.5 11 9.3 - -- 104 88.1 
D.C. 2 4.1 -- -- 8 16.3 3 6.1 36 73.5 
Illinois 35 1.8 57 2.9 263 13.3 34 1.7 1,586 80.3 
Maryland 9 1.9 10 2.1 55 11.3 5 1.0 407 83.7 
Massachusetts 12 2.0 28 4.7 84 14.1 7 1.2 465 78.0 
Mississippi 1 1.3 3 3.8 19 24.4 1 1.3 54 69.2 
New Jersey 47 3.0 79 5.1 201 13.0 21 1.4 1,203 77.6 
New Mexico 38 3.5 54 4.9 160 14.6 13 1.2 833 75.9 
Ohio 11 2.7 25 6.1 64 15.6 3 0.7 307 74.9 
Rhode Island 13 2.1 23 3.7 84 13.5 9 1.5 491 79.2 
All States 204 2.2 349 3.8 1,435 15.7 143 1.6 6,993 76.6 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-35. TA Survey (EOY): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Appendix A: Accessibility Features & Accommodations for Students taking the Paper-Based PARCC Assessments 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 28 1.1 102 4.0 665 26.1 88 3.5 1,662 65.3 
Colorado 4 2.5 4 2.5 29 18.1 2 1.3 121 75.6 
D.C. -- -- -- -- 9 34.6 -- -- 17 65.4 
Illinois 21 1.3 56 3.5 292 18.5 40 2.5 1,173 74.1 
Maryland 12 1.3 30 3.2 172 18.6 29 3.1 682 73.7 
Massachusetts 6 0.9 34 5.3 102 15.8 10 1.5 495 76.5 
Mississippi == -- 1 1.6 23 37.1 3 4.8 35 56.5 
New Jersey 77 2.5 136 4.4 626 20.3 60 1.9 2,186 70.9 
New Mexico 14 4.6 16 5.2 68 22.2 8 2.6 200 65.4 
Ohio 10 2.7 26 7.0 66 17.9 8 2.2 259 70.2 
Rhode Island 2 5.4 -- -- 9 24.3 -- -- 26 70.3 
All States 175 1.8 405 4.1 2,070 21.2 250 2.6 6,880 70.3 


Appendix B: Test Administration Protocol for the Human Reader Accommodations for ELA/L and the Human 


Reader Accessibility Feature for Math 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 30 1.2 98 3.9 639 25.3 81 3.2 1,682 66.5 
Colorado 4 2.5 6 3.8 24 15.0 1 0.6 125 78.1 
D.C. -- -- -- -- 9 34.6 -- -- 17 65.4 
Illinois 21 1.3 64 4.1 291 18.5 33 2.1 1,167 74.0 
Maryland 15 1.6 32 3.5 230 24.8 45 4.9 604 65.2 
Massachusetts 7 1.1 40 6.2 108 16.7 10 1.6 480 74.4 
Mississippi -- -- 1 1.6 20 32.3 3 4.8 38 61.3 
New Jersey 77 2.5 129 4.2 604 19.7 55 1.8 2,201 71.8 
New Mexico 14 4.6 15 5.0 73 24.1 8 2.6 193 63.7 
Ohio 14 3.8 27 7.4 63 17.2 5 1.4 257 70.2 
Rhode Island 2 5.6 1 2.8 6 16.7 -- -- 27 75.0 
All States 185 1.9 414 4.3 2,076 21.3 241 2.5 6,815 70.0 


Appendix C: Protocol for the Use of the Scribe and for Transcribing Student Responses 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 29 1.1 94 3.7 577 22.8 81 3.2 1,749 69.1 
Colorado 4 2.5 4 2.5 27 16.9 1 0.6 124 77.5 
D.C. -- -- 1 3.8 6 23.1 -- -- 19 73.1 
Illinois 20 1.3 55 3.5 274 17.4 32 2.0 1,190 75.7 
Maryland 13 1.4 36 3.9 199 21.6 50 5.4 624 67.7 
Massachusetts 5 0.8 35 5.5 98 15.3 9 1.4 494 77.1 
Mississippi -- -- 1 1.6 19 30.6 3 4.8 39 62.9 
New Jersey 78 2.5 125 4.1 526 17.2 52 1.7 2,282 74.5 
New Mexico 14 4.6 14 4.6 59 19.4 8 2.6 209 68.8 
Ohio 12 3.3 22 6.0 60 16.4 3 0.8 269 73.5 
Rhode Island 1 2.8 -- -- 4 11.1 -- -- 31 86.1 
All States 177 1.8 387 4.0 1,858 19.1 240 2.5 7,054 72.6 
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PARCC Quality of Test Administration 


able G-35. TA Survey (EOY): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Appendix E: Guidance for Selecting and Administering the Extended Time Accommodation 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 32 1.3 103 4.1 940 37.2 142 5.6 1,312 51.9 
Colorado 4 2.5 8 5.0 35 22.0 4 0.6 111 69.8 
D.C. -- -- 1 3.8 14 53.8 2 7.7 9 34.6 
Illinois 20 1.3 71 4.5 481 30.6 65 4.1 935 59.5 
Maryland 13 1.4 33 3.6 329 35.8 78 8.5 466 50.7 
Massachusetts 5 0.8 40 6.2 212 32.9 24 3.7 363 56.4 
Mississippi = -- -- - 25 40.3 4 6.5 33 53.2 
New Jersey 88 2.9 177 5.8 1,065 34.9 120 3.9 1,599 52.4 
New Mexico 14 4.6 19 6.3 92 30.3 10 3.3 169 55.6 
Ohio 12 3.3 26 7.1 133 36.1 15 4.1 182 49.5 
Rhode Island 1 2.9 -- -- 15 42.9 1 2.9 18 51.4 
All States 190 2.0 478 4.9 3,357 34.6 466 4.8 5,212 53.7 
Appendix I: PARCC ELA Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 32 1.3 112 4.4 794 31.5 109 4.3 1,477 58.5 
Colorado 5 3.1 7 4.4 33 20.6 -- - 115 71.9 
D.C. -- -- 1 3.8 10 38.5 -- -- 15 57.7 
Illinois 19 1.2 71 4.5 336 21.4 33 2.1 1,112 70.8 
Maryland 14 1.5 35 3.8 200 21.8 33 3.6 636 69.3 
Massachusetts 6 0.9 50 7.8 126 19.5 14 2.2 449 69.6 
Mississippi -- -- -- -- 22 35.5 3 4.8 37 59.7 
New Jersey 83 2.7 183 6.0 769 25.2 85 2.8 1,931 63.3 
New Mexico 16 5.3 18 5.9 81 26.7 8 2.6 180 59.4 
Ohio 14 3.8 30 8.1 92 24.9 6 1.6 227 61.5 
Rhode Island 2 5.6 -- -- 7 19.4 1 2.8 26 72.2 
All States 192 2.0 508 5.2 2,483 25.6 294 3.0 6,223 64.2 
Appendix J: PARCC Math Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 34 1.3 111 4.4 861 34.2 116 4.6 1,397 55.5 
Colorado 4 2.5 8 5.0 41 25.6 -- -- 107 66.9 
D.C. -- -- 1 3.8 10 38.5 1 3.8 14 53.8 
Illinois 21 1.3 71 4.5 376 24.0 35 2.2 1,062 67.9 
Maryland 13 1.4 34 3.7 216 23.5 33 3.6 622 67.8 
Massachusetts 7 1.1 53 8.3 135 21.1 14 2.2 431 67.3 
Mississippi -- -- -- - 22 35.5 3 4.8 37 59.7 
New Jersey 85 2.8 186 6.1 788 25.8 83 2.7 1,916 62.7 
New Mexico 15 5.0 17 5.6 89 29.6 7 2.3 173 57.5 
Ohio 12 3.3 33 8.9 109 29.5 8 2.2 207 56.1 
Rhode Island 2 5.6 -- -- 9 25.0 1 2.8 24 66.7 
All States 194 2.0 516 5.3 2,670 27.6 304 3.1 6,006 62.0 
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able G-35. TA Survey (EOY): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Assistive Technology Guidelines 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 30 1.2 119 4.7 698 27.9 101 4.0 1,558 62.2 
Colorado 3 1.9 8 5.0 28 17.5 2 1.3 119 74.4 
D.C. -- -- -- -- 6 23.1 -- -- 20 76.9 
Illinois 21 1.3 62 4.0 264 16.9 32 2.0 1,184 75.8 
Maryland 15 1.6 39 4.3 169 18.4 23 2.5 671 73.2 
Massachusetts 6 0.9 45 7.0 82 12.7 7 1.1 506 78.3 
Mississippi -- -- -- -- 20 32.8 3 4.9 38 62.3 
New Jersey 88 2.9 171 5.6 633 20.8 62 2.0 2,092 68.7 
New Mexico 13 4.3 19 6.3 67 22.3 8 2.7 194 64.5 
Ohio 15 4.1 29 8.0 66 18.1 7 1.9 247 67.9 
Rhode Island 2 5.7 1 2.9 4 11.4 -- -- 28 80.0 
All States 194 2.0 493 5.1 2,047 21.2 247 2.6 6,679 69.1 
PARCC Technical Assistance Bulletin - PARCC Assessments and Students with Visual Impairment, Including 
Blindness 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 28 1.1 93 3.7 575 22.9 73 2.9 1,738 69.3 
Colorado 3 1.9 5 3.1 22 13.8 -- -- 130 81.3 
D.C. -- -- -- -- 5 19.2 -- -- 21 80.8 
Illinois 21 1.3 44 2.8 187 12.0 19 1.2 1,289 82.6 
Maryland 11 1.2 28 3.1 119 13.0 16 1.7 744 81.0 
Massachusetts 5 0.8 37 5.8 61 9.5 4 0.6 532 83.3 
Mississippi -- -- -- -- 19 31.1 3 49 39 63.9 
New Jersey 83 Qf 128 4.2 460 15.1 44 1.4 2,337 76.6 
New Mexico 12 4.0 14 4.6 55 18.2 7 2.3 214 70.9 
Ohio 9 2.5 20 5.5 40 11.0 5 1.4 290 79.7 
Rhode Island 2 5.4 -- -- 4 10.8 -- -- 31 83.8 
All States 175 1.8 369 3.8 1,555 16.1 174 1.8 7,389 76.5 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-36. TA Survey (PBA): Q5. The process for identifying accessibility features and accommodations in 


ladvance (via the Personal Needs Profile and associated trainings) was clear and easy to follow. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 94 4.3 490 8.7 822 37.5 56 2.6 1,030 47.0 
Colorado 4 3.3 7 5.7 30 24.4 4 3.3 78 63.4 
D.C. 2 3.8 5 9.6 19 36.5 4 7.7 22 42.3 
Illinois 102 4.9 215 10.4 517 25.1 44 2.1 1,184 57.4 
Maryland 27 5.3 68 13.3 123 24.0 9 1.8 286 55.8 
Massachusetts 34 5.5 116 18.6 143 23.0 7 1.1 322 51.8 
Mississippi 1 1.2 7 8.5 30 36.6 1 1.2 43 52.4 
New Jersey 114 6.9 230 13.9 400 24.1 31 1.9 882 53.2 
New Mexico 69 5.9 149 12.8 292 25.2 13 1.1 637 54.9 
Ohio 31 7.1 64 14.7 109 25.1 3 0.7 228 52.4 
Rhode Island 27 4.1 93 14.0 183 27.5 12 1.8 351 52.7 
All States 506 5.3 1,152 12.0 2,677 27.9 184 1.9 5,077 52.9 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-37. TA Survey (EOY): Q5. The process for identifying accessibility features and accommodations in 


ladvance (via the Personal Needs Profile and associated trainings) was clear and easy to follow. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 66 2.5 231 8.8 965 36.8 60 2.3 1,298 49.5 
Colorado 7 4.1 23 13.5 50 29.2 3 1.8 88 51.5 
D.C. 1 3.7 4 14.8 13 48.1 -- -- 9 33.3 
Illinois 53 3.2 193 11.7 411 25.0 37 2.2 952 57.8 
Maryland 53 5.5 114 11.9 246 25.8 28 2.9 514 53.8 
Massachusetts 29 4.2 104 15.1 153 22.2 10 1.5 393 57.0 
Mississippi 2 2.9 3 4.4 34 50.0 1 1.5 28 41.2 
New Jersey 171 5.3 375 11.6 868 26.7 54 1.7 1,777 54.8 
New Mexico 19 6.1 50 16.2 94 30.4 5 1.6 141 45.6 
Ohio 29 75 76 19.6 74 19.1 3 0.8 206 53.1 
Rhode Island 3 75 5 12.5 11 27.5 -- -- 21 52.5 
All States 437 4.3 1,183 11.6 2,934 28.8 202 2.0 5,440 53.4 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-38. TA Survey (PBA): Q6. Were all the accommodations/accessibility features that were pre-identified 


in the PNP made available to students during a practice session with sample items, tutorial(s), or practice 
est(s)? 


Yes No Not Sure/Don’t Know Not Applicable 
State n % n % n % n % 
Arkansas 730 33.5 179 8.2 727 33.4 543 24.9 
Colorado 30 25.0 15 12.5 30 25.0 45 37.5 
D.C. 22 42.3 6 11.5 12 23.1 12 23.1 
Illinois 474 23.0 244 11.9 621 30.2 720 35.0 
Maryland 126 24.8 63 12.4 159 31.3 160 63.5 
Massachusetts 160 25.7 107 17.2 201 32.3 154 24.8 
Mississippi 12 14.6 4 4.9 33 40.2 33 40.2 
New Jersey 360 21.8 198 12.0 502 30.4 592 35.8 
New Mexico 261 22.6 179 15.5 379 32.8 336 29.1 
Ohio 96 22.3 74 17.2 121 28.1 139 32.3 
Rhode Island 144 22.2 73 11.3 206 31.8 225 34.7 
All States 2,427 25.4 1,144 12.0 3,001 31.5 2,967 31.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-39. TA Survey (EOY): Q6. Were all the accommodations/accessibility features that were pre-identified 


in the PNP made available to students during a practice session with sample items, tutorial(s), or practice 
est(s)? 


Yes No Not Sure/Don’t Know Not Applicable 
State n % n % n % n % 
Arkansas 833 32.0 158 6.1 911 35.0 700 26.9 
Colorado 37 21.6 16 9.4 60 35.1 58 33.9 
D.C. 10 40.0 2 8.0 9 36.0 4 16.0 
Illinois 365 22.3 167 10.2 512 31.2 595 36.3 
Maryland 270 28.3 102 10.7 307 32.2 274 28.8 
Massachusett 
s 158 23.1 91 13.3 222 32.5 213 31.1 
Mississippi 16 23.2 3 4.3 24 34.8 26 37.7 
New Jersey 798 24.7 333 10.3 937 29.0 1,168 36.1 
New Mexico 65 21.2 43 14.0 104 33.9 95 30.9 
Ohio 107 27.4 60 15.3 112 28.6 112 28.6 
Rhode Island 12 31.6 3 7.9 9 23.7 14 36.8 
All States 2,685 26.5 980 9.7 3,218 31.7 3,266 32.2 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able G-40. TA Survey (PBA): Q7. During administration, were changes made to a student’s PNP or to the 


availability of accommodations or accessibility features? 


Yes No Not Sure/Don’t Know Not Applicable 
State n % n % n % n % 
Arkansas 68 3.1 823 37.5 759 34.6 544 24.8 
Colorado 10 8.1 35 28.2 34 27.4 45 36.3 
D.C. 7 13.5 19 36.5 13 25.0 13 25.0 
Illinois 101 4.9 657 31.8 606 29.3 703 34.0 
Maryland 29 5.7 159 31.1 172 33.7 151 29.5 
Massachusett 
s 49 7.9 226 36.4 198 31.9 148 23.8 
Mississippi 3 3.6 21 25.3 22 26.5 37 44.6 
New Jersey 74 4.5 515 31.0 486 29.3 584 35.2 
New Mexico 45 3.9 407 35.1 378 32.6 331 28.5 
Ohio 25 5.8 154 35.8 122 28.4 129 30.0 
Rhode Island 15 2.3 259 39.2 191 28.9 195 29.5 
All States 427 4.5 3,292 34.3 2,989 31.2 2,885 30.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-41. TA Survey (EOY): Q7. During administration, were changes made to a student’s PNP or to the 


availability of accommodations or accessibility features? 


Yes No Not Sure/Don’t Know Not Applicable 
State n % n % n % n % 
Arkansas 93 3.6 926 35.4 930 35.6 666 25.5 
Colorado 8 4.8 44 26.5 65 39.2 49 29.5 
D.C. 1 3.7 11 40.7 10 37.0 5 18.5 
Illinois 85 5.2 510 31.0 493 30.0 556 33.8 
Maryland 51 5.3 319 33.4 326 34.1 260 27.2 
Massachusetts 39 5.7 216 31.3 232 33.7 202 29.3 
Mississippi 1 1.4 23 33.3 21 30.4 24 34.8 
New Jersey 145 4.5 1,048 32.2 892 27.4 1,168 35.9 
New Mexico 18 5.8 101 32.6 101 32.6 90 29.0 
Ohio 26 6.7 146 37.5 108 27.8 109 28.0 
Rhode Island -- -- 19 47.5 8 20.0 13 32.5 
All States 469 4.6 3,380 33.2 3,197 31.4 3,148 30.9 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Sample Items/Tutorials 


Table G-42. TA Survey (PBA): Q1. How did students in your session(s) practice with PARCC content prior to 


administration? Select all that apply.” 


Not sure/Don’t 


Sample Items Tutorial Practice Test Did Not Practice Know 

State n % n % n % n % n % 

Arkansas 1,558 66.1 1,260 53.4 1,540 65.3 36 1.5 301 12.8 
Colorado 82 60.3 43 31.6 69 50.7 6 4.4 27 19.9 
D.C. 35 63.6 25 45.5 29 52.7 -- -- 13 23.6 
Illinois 1,482 65.5 1,295 57.2 1,281 56.6 84 3.7 218 9.6 
Maryland 348 62.7 259 46.7 315 56.8 6 1.1 83 15.0 
Massachusetts 483 69.7 388 56.0 465 67.1 19 2.7 55 7.9 
Mississippi 52 56.5 28 30.4 48 52.2 2 2.2 19 20.7 
New Jersey 1,202 62.6 899 46.8 1,132 59.0 33 1.7 271 14.1 
New Mexico 866 68.9 571 45.5 886 70.5 40 3.2 110 8.8 
Ohio 326 70.4 248 53.6 302 65.2 10 2.2 64 13.8 
Rhode Island 522 72.7 402 56.0 463 64.5 14 1.9 65 9.1 
All States® 6,978 66.2 5,435 51.5 6,549 62.1 251 2.4 1,231 11.7 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
’N-counts include individuals with missing data for state location who selected a response. 


Table G-43. TA Survey (EOY): Q1. How did students in your session(s) practice with PARCC content prior to 


administration? Select all that apply.? 
Not sure/Don’t 


Sample Items Tutorial Practice Test Did Not Practice Know 

State n % n % n % n % n % 

Arkansas 1,766 62.2 1,403 49.4 1,699 59.9 51 1.8 451 15.9 
Colorado 102 55.7 73 39.9 78 42.6 13 7.1 33 18.0 
D.C. 15 50.0 13 43.3 16 53.3 2 6.7 5 16.7 
Illinois 1,229 68.2 968 53.7 1,050 58.2 75 4.2 150 8.3 
Maryland 649 61.5 548 51.9 567 53.7 28 2.7 114 10.8 
Massachusetts 517 68.8 397 52.8 492 65.4 26 3.5 59 7.8 
Mississippi 40 55.6 26 36.1 31 43.1 4 5.6 21 29.2 
New Jersey 2,408 66.0 1,861 51.0 2,172 59.6 111 3.0 479 13.1 
New Mexico 225 68.2 158 47.9 218 66.1 13 3.9 34 10.3 
Ohio 299 69.1 222 51.3 274 63.3 11 2.5 42 9.7 
Rhode Island 36 81.8 32 72.7 34 77.3 1 2.3 1 2.3 
All States? 7,316 65.1 5,723 51.0 6,656 59.3 335 3.0 1,393 12.4 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
’N-counts include individuals with missing data for state location who selected a response. 
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Table G-44. TA Survey (PBA): Q2. How did you, as a Test Administrator, work with PARCC content prior to 


administration? Select all that apply.? 


Sample Items Tutorial Practice Test Did Not Practice 
State n % n % n % n % 
Arkansas 1,622 68.8 1,576 66.8 1,327 56.3 223 9.5 
Colorado 88 64.7 66 48.5 85 62.5 14 10.3 
D.C. 37 67.3 32 58.2 30 54.5 8 14.5 
Illinois 1,671 73.8 1,587 70.1 1,392 61.5 139 6.1 
Maryland 374 67.4 322 58.0 274 49.4 69 12.4 
Massachusetts 504 72.7 495 71.4 454 65.5 45 6.5 
Mississippi 48 52.2 43 46.7 34 37.0 17 18.5 
New Jersey 1,339 69.7 1,262 65.7 1,164 60.6 130 6.8 
New Mexico 905 72.1 743 59.2 812 64.6 97 7.7 
Ohio 345 74.5 347 74.9 296 63.9 27 5.8 
Rhode Island 536 74.7 494 68.8 452 63.0 53 7.4 
All States? 7,494 71.1 6,990 66.3 6,338 60.1 825 7.8 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
’N-counts include individuals with missing data for state location who selected a response. 


Table G-45. TA Survey (EOY): Q2. How did you, as a Test Administrator, work with PARCC content prior to 


administration? Select all that apply.” 


Sample Items Tutorial Practice Test Did Not Practice 
State n % n % n % n % 
Arkansas 1,842 64.9 1,816 64.0 1,491 52.5 316 11.1 
Colorado 119 65.0 92 50.3 86 47.0 25 13.7 
D.C. 19 63.3 18 60.0 19 63.3 3 10.0 
Illinois 1,314 72.9 1,254 69.6 1,082 60.0 105 5.8 
Maryland 660 62.5 618 58.5 506 47.9 133 12.6 
Massachusetts 529 70.3 503 66.9 482 64.1 55 7.3 
Mississippi 39 54.2 44 61.1 26 36.1 15 20.8 
New Jersey 2,634 72.2 2,458 67.4 2,241 61.4 244 6.7 
New Mexico 235 71.2 202 61.2 218 66.1 27 8.2 
Ohio 325 75.1 312 72.1 273 63.0 17 3.9 
Rhode Island 36 81.8 31 70.5 29 65.9 -- -- 
All States? 7,781 69.3 7,375 65.7 6,471 57.6 945 8.4 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
’N-counts include individuals with missing data for state location who selected a response. 
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able G-46. TA Survey (PBA): Q3. Completion of the PARCC tutorial helped me to better understand the tools 


land functionalities of the TestNav system. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 95 4.3 182 8.3 1,386 63.1 324 14.7 210 9.6 
Colorado 4 3.2 13 10.5 70 56.5 10 8.1 27 21.8 
D.C. 2 3.8 1 1.9 33 63.5 5 9.6 11 21.2 
Illinois 95 4.5 195 9.3 1,220 58.4 350 16.8 228 10.9 
Maryland 19 3.7 51 9.9 295 57.3 54 10.5 96 18.6 
Massachusetts 31 4.9 81 12.9 385 61.2 77 12.2 55 8.7 
Mississippi 1 1.2 10 12.2 46 56.1 8 9.8 17 20.7 
New Jersey 101 6.0 248 14.8 1,000 59.5 182 10.8 149 8.9 
New Mexico 81 6.9 165 14.1 679 57.9 84 7.2 163 13.9 
Ohio 18 4.1 64 14.7 274 62.8 42 9.6 38 8.7 
Rhode Island 28 4.2 89 13.3 393 58.8 69 10.3 89 13.3 
All States 476 4.9 1,100 11.4 5,800 60.0 1,210 12.5 1,085 11.2 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able G-47. TA Survey (EOY): Q3. Completion of the PARCC tutorial helped me to better understand the tools 


land functionalities of the TestNav system. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 114 4.3 205 7.8 1,713 64.9 327 12.4 279 10.6 
Colorado 1 0.6 18 10.8 101 60.5 18 10.8 29 17.4 
D.C. -- -- 5 18.5 14 51.9 5 18.5 3 11.1 
Illinois 46 2.8 140 8.4 1,071 64.6 237 14.3 163 9.8 
Maryland 47 4.9 90 9.3 549 57.0 139 14.4 138 14.3 
Massachusetts 24 3.5 86 12.5 441 64.0 75 10.9 63 9.1 
Mississippi 2 2.9 8 11.6 42 60.9 5 7.2 12 17.4 
New Jersey 151 4.6 416 12.6 2,046 62.2 362 11.0 315 9.6 
New Mexico 20 6.4 43 13.7 192 61.1 16 5.1 43 13.7 
Ohio 20 5.1 67 17.2 238 61.0 31 7.9 34 8.7 
Rhode Island 1 2.5 3 7.5 30 75.0 3 7.5 3 7.5 
All States 426 4.1 1,083 10.5 6,461 62.9 1,222 11.9 1,087 10.6 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Appendix H — Item-level Results from Test Administrator Survey: 
PBT Administration with Breakouts for PBA and EOY 


Paper-Based 


Background Information 


Table H-1. TA Survey Respondents by Administration and State 


PBT PBA PBT EOY 
N-count % N-count % 

Arkansas 89 5.1 189 14.5 
Colorado 18 1.0 11 0.8 
District of Columbia 2 0.1 2 0.2 
Illinois 440 25.3 160 12.3 
Maryland 26 1.5 154 11.8 
Massachusetts 662 38.0 449 34.5 
Mississippi 40 2.3 = =- 

New Jersey 31 1.8 173 13.3 
New Mexico 57 3.3 44 3.4 
Ohio 239 13.7 94 7.2 
Rhode Island 136 7.8 25 1.9 
All States 2,336° 100.0 1,469° 100.0 


Note. Percentages are based on the total number of valid responses for each state. Number of 
missing for PBT PBA = 596. Number of missing for PBT EOY = 168. 

"Includes the 596 cases that did not indicate their state. 

Includes the 168 cases that did not indicate their state. 
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Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 N/A 

State n % n % n % n % n % n % n % n % n % n % 

Arkansas 15 69 11 12.4 22 24.7 10 11.2 10 11.2 8 9.0 14 15.7 13 (14.6 3 34 9 10.1 
Colorado 3 16.7 6 33.3 1 5.6 1 56 1 5.6 1 5.6 2 11.1 2 11.1 3 16.7 2 11.1 
D.C. -- -- 2 100.0 -- -- -- -- -- -- -- -- -- -- -- -- -- -- “oe 
Illinois 131 29.8 143 32.5 105 23.9 56 12.7 41 9.3 44 10.0 13 33.0 -- - 3 O07 19 43 
Maryland 10 38.5 7 26.9 11 42.3 -- -- -- - -- -- 1 3.8 3 11.5 -- -- 2 7.7 
Massachusetts 158 23.9 169 25.5 194 29.3 117 17.7. 131 19.8 127 19.2 7 1.1 2 0.3 6 09 24 3.6 
Mississippi 8 20.0 6 15.0 6 15.0 -- - 3 7.5 4 10.0 -- -- 9 22.5 -- -- 8 20.0 
New Jersey 4 12.9 1 3.2 4 12.9 6 19.4 7 22.6 9 29.0 2 6.5 3 9.7 3 9.7 1 3.2 
New Mexico 12 21.1 11 19.3 5 88 17 29.8 11 19.3 12 21.1 -- -- 3 5.3 6 105 4 7.0 
Ohio 21 +88 70 29.3 52 21.8 26 10.9 24 10.0 23 («9.6 34 14.2 3 1.3 1 04 67 28.0 
Rhode Island 31 22.8 26 19.1 24 17.6 23 16.9 21 15.4 19 14.0 1 0.7 1 0.7 - -- 3 2.2 
All States 590 25.3 630 27.0 568 24.3 294 12.6 279 11.9 272 11.6 76 = 33.3 39 1.7 25 1.1 143 6.1 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
>N-counts include individuals with missing data for state location who selected a grade level. 
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Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 N/A 

State n % n % n % n % n % n % n % n % n % n % 

Arkansas 32 16.9 29 15.3 30 15.9 34 18.0 18 9.5 24 12.7 14 7.4 14 7.4 9 48 20 10.6 
Colorado 7 63.6 1 9.1 4 36.4 -- -- -- -- -- -- -- -- -- -- -- -- 1 9.1 
D.C. 1 50.0 2 100.0 - -- -- - -- -- - - -- -- -- -- -- -- -- -- 
Illinois 32 20.0 32 20.0 24 15.0 23 14.4 22 13.8 19 11.9 17 10.6 2 1.3 5 3.1 13 8.1 
Maryland 36 23.4 30 19.5 29 18.8 31 20.1 24 15.6 25 16.2 -- -- 10 6.5 - -- 11 «(7.1 
Massachusetts 148 33.0 120 26.7 138 30.7 78 17.4 44 9.8 66 14.7 10 2.2 1 0.2 9 2.0 16 3.6 
New Jersey 31 17.9 38 22.0 41 23.7 23 13.3 31 17.9 35 20.2 1 0.6 1 0.6 1 0.6 3 17 
New Mexico 12 27.3 14 31.8 13 29.5 10 22.7 7 15.9 8 18.2 3 68 7 15.9 5 11.4 1 2.3 
Ohio 4 43 18 19.1 27 «28.7 23 24.5 8 8.5 15 16.0 3 3.2 -- - - -- 19 20.2 
Rhode Island 6 24.0 10 40.0 1 4.0 -- - 5 20.0 4 16.0 - - -- -- -- -- - = 

All States” 354 24.1 350 23.8 357 24.3 235 160 165 11.2 204 13.9 48 3.3 35 2.4 29 2.0 84 5.7 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
>N-counts include individuals with missing data for state location who selected a grade level. 
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Grade3 Grade4 Grade5 Grade6 Grade7 Grade8 ALG 1 GEO ALG 2 INT 1 INT 2 INT 3 N/A 
State n % n % n % n % n % n % n % n % n % n % n % n % n % 
Arkansas 1415.7 1112.4 22 24.77 11 12.4 #10 11.2 10 11.2 12 135 13 146 4 45 sos - - -- 4 45 
Colorado 2 11.1 6 33.3 1 5.6 1 5.6 1 5.6 1 5.6 1 5.6 1 5.6 4 22.2 -- -- -- -- -- -- 2 11.1 
Illinois 129 29.3 144 32.7 104 236 55 125 40 91 40 91 5 114 - -- 5 1.1 3 07 - = 1 O02 22 5.0 
Maryland 10 38.5 9 34.6 7 26.9 138 - - so 2 #77 = -- 1 3.8 so - -- -- 3 11.5 
Massachusetts 156 23.6 169 25.5 187 28.2 114 17.2 130 19.6 121 183 19 29 2 03 5 O8 102 - = - - 30 4.5 
Mississippi 7 17.5 7 17.5 5125 - - 2 50 3 75 #8 20.0 - -- 1 2.5 so - -- -- 8 20.0 
New Jersey 4 12.9 1 3.2 3 9.7 5 161 6194 9 290 2 65 5 161 1 3.2 sos oe -- -- 2 6.5 
New Mexico 13 22.8 10 17.5 5 88 12 21.1 7 12.3 6105 4 70 2 35 3 5.3 1 $18 3 53 - -- 3 5.3 
Ohio 112 46.9 66 276 4619.2 21 88 20 84 24 100 26 109 12 50 2 08 7 29 2 08 2 O08 14 5.9 
Rhode Island 31 22.8 25 18.4 23 169 21 154 21 154 15 11.0 6 4.4 1 O77 - - -- -- - -- - -- 4 2.9 
All States” 669 28.6 633 27.1 546 23.4 277 11.9 267 11.4 256 11.0 87 3.7 36 15 26 11 12 05 5 O02 3 O01 101 4.3 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 


>N-counts include individuals with missing data for state location who selected a grade level. 
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Grade3 Grade4 Grade5 Grade6 Grade7 Grade8 ALG 1 GEO ALG 2 INT 1 INT 2 INT3 N/A 
State n % n % n % n % n % n % n % n % n % n % n % n % n % 
Arkansas 33 175 31 164 31164 34 180 18 95 23 12.2 15 79 15 7.9 10 5.3 -- -- oe -- -- 15 7.9 
Colorado 7 63.6 2 18.2 4 36.4 oo -- -- oe - -- -- - = oo -o- -- -- - = 
D.C. 1 50.0 2100.0 -- -- so - so soos -- -- os cos mo -- -- soo 
Illinois 37 23.1 30 206 25156 23 144 21 131 19 119 2 13 1 06 10 6.3 mo 1 06 - -- 17 10.6 
Maryland 38 24.7 30 195 3019.5 33 21.4 18 11.7 16 104 33 214 -- -- 8 5.2 so so -- -- 3 19 
Massachusetts 147 32.7 119 26.55 142 316 79 176 47 105 72 160 12 27 2 04 7 1.6 so co -- - 11 2.4 
New Jersey 31 17.9 38 22.0 42 24.3 22 12.7 32 185 22 12.77 13 75 4 2.3 - = sos so -- -- 4 2.3 
New Mexico 12 27.3 15 341 13295 10 22.77 7 159 10 227 #5 114 4 91 3 68 5114 1 23 - -- soo 
Ohio 25 266 20 21.3 2829.8 23 245 8 85 13 138 7 74 3 32 - = so os -- -- 4 43 
Rhode Island 6 24.0 11 44.0 2 80 - - 5 20.0 4 160 - - -- -- oo sos so -- -- oo 
All States 380 25.9 358 24.4 365 24.8 237 16.1 162 11.0 187 12.7 87 59 29 2.0 38 2.6 5 03 2 01 - - 58 3.9 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
>N-counts include individuals with missing data for state location who selected a grade level. 
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PARCC 


Test Administrator's Training Experience 


Quality of Test Administration 


able H-6. TA Survey (PBA): Q1. Please indicate your level of agreement with the following statement: “This 


raining effectively prepared me to administer the PARCC assessments.” 
Introduction to PARCC training modules 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 5 5.7 14 15.9 39 43.8 19 21.3 11 12.4 
Colorado 2 11.1 1 5.6 7 38.9 3 16.7 5 27.8 
D.C. -- -- -- -- 1 50.0 1 50.0 -- -- 
Illinois 22 5.1 51 11.8 196 45.3 62 14.3 102 23.6 
Maryland 1 4.0 -- -- 13 52.0 6 24.0 5 20.0 
Massachusetts 23 3.6 65 10.1 326 50.7 96 14.9 133 20.7 
Mississippi 3 7.5 5 12.5 13 32.5 12 30.0 7 17.5 
New Jersey -- -- 5 17.2 18 62.1 4 13.8 2 6.9 
New Mexico 3 5.5 5 9.1 34 61.8 6 10.9 7 12.7 
Ohio 31 13.1 38 16.0 86 36.3 19 8.0 63 26.6 
Rhode Island 8 6.2 24 18.5 67 51.5 9 6.9 22 16.9 
All States 129 5.7 253 11.1 1,064 46.6 348 15.2 488 21.4 
Accessibility Features & Accommodations Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 8 9.2 11 12.6 43 49.4 12 13.8 13 14.9 
Colorado 2 11.1 2 11.1 6 33.3 3 16.7 5 27.8 
D.C. -- -- -- -- 1 50.0 1 50.0 -- -- 
Illinois 22 5.1 62 14.4 173 40.2 54 12.6 119 27.7 
Maryland 1 4.0 1 4.0 9 36.0 6 24.0 8 32.0 
Massachusetts 27 4.2 88 13.7 294 45.7 72 11.2 163 25.3 
Mississippi 2 5.0 4 10.0 11 27.5 11 27.5 12 30.0 
New Jersey 2 6.9 4 13.8 14 48.3 5 17.2 4 13.8 
New Mexico 3 5.6 7 13.0 31 57.4 4 7.4 9 16.7 
Ohio 27 11.5 46 19.7 79 33.8 15 6.4 67 28.6 
Rhode Island 11 8.5 27 20.8 59 45.4 8 6.2 25 19.2 
All States 131 5.8 297 13.1 979 43.1 301 13.2 564 24.8 


Administration of Paper-Based Assessments for Test Administrators Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 5 5.7 10 11.5 43 49.4 16 18.4 13 14.9 
Colorado 2 11.1 -- -- 3 16.7 2 11.1 11 61.1 
D.C. -- -- -- -- 1 50.0 1 50.0 -- -- 
Illinois 23 5.4 42 9.8 199 46.4 77 17.9 88 20.5 
Maryland 1 4.0 -- -- 12 48.0 5 20.0 7 28.0 
Massachusetts 20 3.1 60 9.4 350 54.7 98 15.3 112 17.5 
Mississippi 5 12.8 4 10.3 12 30.8 13 33.3 5 12.8 
New Jersey -- = 8 26.7 11 36.7 5 16.7 6 20.0 
New Mexico 2 3.6 6 10.9 34 61.8 4 7.3 9 16.4 
Ohio 25 10.8 36 15.6 94 40.7 21 9.1 55 23.8 
Rhode Island 8 6.2 19 14.6 74 56.9 7 5.4 22 16.9 
All States 122 5.4 231 10.2 1,092 48.1 372 16.4 451 19.9 
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PARCC Quality of Test Administration 


able H-6. TA Survey (PBA): Q1. Please indicate your level of agreement with the following statement: “This 


raining effectively prepared me to administer the PARCC assessments.” 
PearsonAccess"™™ Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 5 5.7 7 8.0 44 50.6 10 11.5 21 24.1 
Colorado 2 11.1 1 5.6 6 33.3 2 11.1 7 38.9 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 26 6.1 54 12.7 124 29.2 36 8.5 184 43.4 
Maryland 1 4.0 -- -- 10 40.0 3 12.0 11 44.0 
Massachusetts 23 4.1 70 11.0 196 30.7 43 6.7 304 47.6 
Mississippi 3 7.7 4 10.3 10 25.6 9 23.1 13 33.3 
New Jersey 1 3.4 8 27.6 11 37.9 7 24.1 2 6.9 
New Mexico 4 7.5 6 11.3 27 50.9 4 7.5 12 22.6 
Ohio 29 12.4 39 16.7 52 22.2 12 5.1 102 43.6 
Rhode Island 10 7.6 22 16.8 44 33.6 4 3.1 51 38.9 
All States 136 6.0 261 11.6 685 30.4 182 8.1 987 43.8 
Student Registration Import Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 5 5.7 9 10.2 37 42.0 8 9.1 29 33.0 
Colorado 2 11.1 -- -- 5 27.8 2 11.1 9 50.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 27 6.4 46 10.9 75 17.8 31 7.3 243 57.6 
Maryland 1 4.0 -- -- 4 16.0 2 8.0 18 72.0 
Massachusetts 29 4.6 64 10.2 159 25.3 31 49 345 54.9 
Mississippi 3 7.7 4 10.3 10 25.6 8 20.5 14 35.9 
New Jersey -- -- 7 23.3 7 23.3 3 10.0 13 43.3 
New Mexico 3 5.9 6 11.8 25 49.0 2 3.9 15 29.4 
Ohio 36 15.4 33 14.1 30 12.8 7 3.0 128 54.7 
Rhode Island 11 8.6 16 12.5 28 21.9 2 1.6 71 55.5 
All States 144 6.4 233 10.4 531 23.8 140 6.3 1,185 53.1 
Student Readiness Resources for PARCC Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 10.3 7 8.0 45 51.7 8 9.2 18 20.7 
Colorado 2 11.1 2 11.1 5 27.8 2 11.1 7 38.9 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 31 7.3 67 15.7 118 27.6 38 8.9 173 40.5 
Maryland 1 4.0 -- -- 8 32.0 3 12.0 13 52.0 
Massachusetts 40 6.4 79 12.6 175 27.8 42 6.7 293 46.6 
Mississippi 3 7.7 4 10.3 10 25.6 10 25.6 12 30.8 
New Jersey 1 3.3 6 20.0 11 36.7 4 13.3 8 26.7 
New Mexico 4 7.7 7 13.5 28 53.8 3 5.8 10 19.2 
Ohio 41 17.7 37 15.9 46 19.8 5 2.2 103 44.4 
Rhode Island 10 7.9 18 14.2 54 42.5 4 3.1 41 32.3 
All States 178 8.0 280 12.6 679 30.4 170 7.6 924 41.4 


(continued) 
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PARCC Quality of Test Administration 


able H-6. TA Survey (PBA): Q1. Please indicate your level of agreement with the following statement: “This 


raining effectively prepared me to administer the PARCC assessments.” 
Training from my state department of education 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 10.5 12 14.0 35 40.7 8 9.3 22 25.6 
Colorado 5 27.8 -- -- 3 16.7 Hl 5.6 9 50.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 58 13.6 72 16.9 81 19.1 27 6.4 187 44.0 
Maryland 1 4.0 3 12.0 6 24.0 4 16.0 11 44.0 
Massachusetts 62 9.7 99 15.6 146 23.0 31 4.9 298 46.9 
Mississippi 3 7.5 9 22.5 9 22.5 5 12.5 14 35.0 
New Jersey 3 10.0 7 23.3 7 23.3 1 3.3 12 40.0 
New Mexico 10 19.2 7 13.5 22 42.3 1 1.9 12 23.1 
Ohio 56 24.1 44 19.0 29 16.4 4 1.7 90 38.8 
Rhode Island 17 13.1 25 19.2 34 26.2 3 2.3 51 39.2 
All States 279 12.4 344 15.3 553 24.7 134 6.0 932 41.6 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 8 9.2 4 4.6 44 50.6 16 18.4 15 17.2 
Colorado 4 22.2 2 11.1 3 16.7 1 5.6 8 44.4 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 32 7.4 46 10.6 166 38.4 91 21.1 97 22.5 
Maryland 1 4.0 2 8.0 10 40.0 5 20.0 7 28.0 
Massachusetts 40 6.2 81 12.6 246 38.4 90 14.0 184 28.7 
Mississippi 3 7.5 3 7.5 15 37.5 16 40.0 3 7.5 
New Jersey 1 3.3 3 10.0 18 60.0 7 23.3 1 3.3 
New Mexico 5 9.1 5 9.1 31 56.4 4 7.3 10 18.2 
Ohio 28 12.0 39 16.7 100 42.7 32 13.7 35 15.0 
Rhode Island 9 6.9 26 19.8 53 40.5 10 7.6 33 25.2 
All States 165 7.2 266 11.7 891 39.1 400 17.6 554 24.3 
Training from my school 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 6 6.8 5 5.7 49 55.7 23 26.1 5 5.7 
Colorado 1 5.6 2 11.1 10 55.6 5 27.8 -- -- 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 19 4.4 30 6.9 213 49.2 127 29.3 44 10.2 
Maryland 2 8.0 dl. 4.0 10 40.0 11 44.0 1 4.0 
Massachusetts 26 4.0 51 7.8 343 52.4 213 32.6 21 3.2 
Mississippi 3 7.5 4 10.0 13 32.5 19 47.5 1 2.5 
New Jersey - -- 3 9.7 19 61.3 8 25.8 dl 3.2 
New Mexico 4 7.0 1 1.8 34 59.6 16 28.1 2 3.5 
Ohio 27 11.4 30 12.7 114 48.3 43 18.2 22 9.3 
Rhode Island 7 5.3 13 9.9 80 61.1 22 16.8 9 6.9 
All States 129 5.6 167 7.2 1,170 50.7 723 31.4 117 5.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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PARCC Quality of Test Administration 


able H-7. TA Survey (EOY): Q1. Please indicate your level of agreement with the following statement: “This 


raining effectively prepared me to administer the PARCC assessments.” 
Introduction to PARCC training modules 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 4.8 20 10.8 111 59.7 29 15.6 17 9.1 
Colorado -- -- 1 9.1 8 72.7 2 18.2 -- -- 
D.C. -- - - - 2 100.0 -- -- - -- 
Illinois 6 3.8 26 16.7 75 48.1 15 9.6 34 21.8 
Maryland 6 3.9 9 5.9 77 50.7 23 15.1 37 24.3 
Massachusetts 21 4.8 41 9.4 233 53.3 74 16.9 68 15.6 
New Jersey 16 9.7 16 9.7 85 51.5 22 13.3 26 15.8 
New Mexico 1 2.3 4 9.3 27 62.8 4 9.3 7 16.3 
Ohio 6 6.8 13 21.6 37 63.6 5 5.7 27 30.7 
Rhode Island -- -- 3 12.0 16 64.0 1 4.0 5 20.0 
All States 69 4.8 138 9.7 767 53.7 214 15.0 241 16.9 
Accessibility Features & Accommodations Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 11 5.9 19 10.2 104 55.9 23 12.4 29 15.6 
Colorado - -- 1 10.0 6 60.0 2 20.0 1 10.0 
D.C. - -- - - 2 100.0 -- -- - -- 
Illinois 7 4.6 31 20.4 63 41.4 9 5.9 42 27.6 
Maryland 9 6.0 10 6.7 68 45.3 23 15.3 40 26.7 
Massachusetts 27 6.2 61 14.0 202 46.3 54 12.4 92 21.1 
New Jersey 14 8.6 23 14.1 74 45.4 20 12.3 32 19.6 
New Mexico 2 4.7 5 11.6 24 55.8 4 9.3 8 18.6 
Ohio 5 5.6 15 16.7 34 37.8 5 5.6 31 34.4 
Rhode Island -- -- 3 13.0 13 56.5 -- -- 7 30.4 
All States 81 5.7 173 12.2 679 47.9 178 12.5 308 21.7 
Administration of Paper-Based Assessments for Test Administrators Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 4.8 15 8.0 112 59.9 26 13.9 25 13.4 
Colorado -- -- 1 9.1 6 54.5 4 36.4 -- -- 
D.C. - -- - - 2 100.0 -- -- - -- 
Illinois 7 4.6 23 15.0 72 47.1 17 11.1 34 22.2 
Maryland 7 4.6 11 7.2 75 49.0 30 19.6 30 19.6 
Massachusetts 23 5.3 40 9.1 238 54.3 77 17.6 60 13.7 
New Jersey 14 8.5 10 6.1 52 31.7 12 7.3 76 46.3 
New Mexico 2 4.8 3 7.1 22 52.4 5 11.9 10 23.8 
Ohio 7 7.8 16 17.8 39 43.3 6 6.7 22 24.4 
Rhode Island -- -- 2 8.3 17 70.8 -- -- 5 20.8 
All States 75 5.3 125 8.8 724 50.7 221 15.5 283 19.8 


(continued) 
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Quality of Test Administration 


able H-7. TA Survey (EOY): Q1. Please indicate your level of agreement with the following statement: “This 


raining effectively prepared me to administer the PARCC assessments.” 
PearsonAccess"™™ Training Module 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 8 4.3 23 12.4 83 44.6 21 11.3 51 27.4 
Colorado -- -- 1 9.1 8 72.7 -- -- 2 18.2 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 8 5.2 26 17.0 53 34.6 6 3.9 60 39.2 
Maryland 8 5.3 16 10.5 40 26.3 8 5.3 80 52.6 
Massachusetts 25 5.8 51 11.8 120 27.7 29 6.7 208 48.0 
New Jersey 14 8.7 20 12.4 60 37.3 16 9.9 51 31.7 
New Mexico 2 4.7 5 11.6 21 48.8 4 9.3 11 25.6 
Ohio 5 5.7 16 18.4 25 28.7 5 5.7 36 41.4 
Rhode Island -- -- 2 8.7 11 47.8 -- -- 10 43.5 
All States 77 5.5 169 12.0 487 34.5 107 7.6 571 40.5 
Student Registration Import Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 10 5.4 21 11.4 72 39.1 18 9.8 63 34.2 
Colorado -- -- 1 10.0 4 40.0 -- -- 5 50.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 11 7.1 25 16.2 38 24.7 5 3.2 75 48.7 
Maryland 8 5.3 14 9.3 25 16.7 6 4.0 97 64.7 
Massachusetts 27 6.3 46 10.7 96 22.4 21 4.9 239 55.7 
New Jersey 13 8.2 14 8.8 43 27.0 9 5.7 80 50.3 
New Mexico 1 2.4 4 9.5 18 42.9 3 7.1 16 38.1 
Ohio 6 6.9 14 16.1 16 18.4 3 3.4 48 55.2 
Rhode Island -- -- 2 8.7 2 8.7 -- -- 19 82.6 
All States 82 5.9 147 10.5 375 26.8 81 5.8 714 51.0 
Student Readiness Resources for PARCC Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 8 4.4 17 9.3 88 48.1 20 10.9 50 27.3 
Colorado -- -- 1 10.0 5 50.0 -- -- 4 36.4 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 9 6.0 28 18.5 42 27.8 7 4.6 65 43.0 
Maryland 8 5.3 15 9.9 39 25.8 9 6.0 80 53.0 
Massachusetts 32 7.4 53 12.3 118 27.4 31 7.2 197 45.7 
New Jersey 16 10.3 20 12.8 57 36.5 12 7.7 51 32.7 
New Mexico 2 4.7 6 14.0 21 48.8 4 9.3 10 23.3 
Ohio 7 7.9 15 16.9 21 23.6 2 2.2 44 49.4 
Rhode Island 1 4.3 1 4.3 9 39.1 -- -- 12 52.2 
All States 89 6.3 167 11.9 465 33.2 107 7.6 574 40.9 
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PARCC Quality of Test Administration 


able H-7. TA Survey (EOY): Q1. Please indicate your level of agreement with the following statement: “This 


raining effectively prepared me to administer the PARCC assessments.” 
Training from my state department of education 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 12 6.5 22 12.0 73 39.7 16 8.7 61 33.2 
Colorado 2 20.0 1 10.0 3 30.0 -- -- 4 40.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 23 14.9 30 19.5 35 22.7 2 1.3 64 41.6 
Maryland 16 10.6 22 14.6 28 18.5 9 6.0 76 50.3 
Massachusetts 44 10.1 76 17.5 99 22.8 21 4.8 195 44.8 
New Jersey 29 18.2 29 18.2 37 23.3 10 6.3 54 34.0 
New Mexico 2 4.7 12 27.9 15 34.9 3 7.0 11 25.6 
Ohio 13 14.9 18 20.7 19 21.8 1 1.1 36 41.4 
Rhode Island 3 12.5 1 4.2 4 16.7 -- -- 16 66.7 
All States 155 11.0 225 15.9 382 27.1 81 5.7 568 40.3 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 10 5.4 19 10.2 97 52.2 38 20.4 22 11.8 
Colorado 2 18.2 -- -- 5 45.5 3 27.3 i 9.1 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 14 9.1 18 11.7 69 44.8 18 11.7 35 22.7 
Maryland 11 7.2 22 14.5 52 34.2 13 8.6 54 35.5 
Massachusetts 21 4.8 59 13.5 181 41.4 70 16.0 106 24.3 
New Jersey 14 8.5 18 11.0 73 44.5 49 29.9 10 6.1 
New Mexico 2 4.7 5 11.6 22 51.2 7 16.3 7 16.3 
Ohio 8 8.7 18 19.6 43 46.7 11 12.0 12 13.0 
Rhode Island 1 4.2 3 12.5 8 33.3 -- -- 12 50.0 
All States 91 6.4 170 11.9 622 43.5 256 17.9 290 20.3 
Training from my school 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 11 5.9 12 6.5 108 58.1 49 26.3 6 3.2 
Colorado -- -- -- -- 3 27.3 7 63.6 -- -- 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 12 7.8 15 9.7 83 53.9 31 20.1 13 8.4 
Maryland 8 5.2 6 3.9 78 51.0 57 37.3 4 2.6 
Massachusetts 13 2.9 37 8.4 258 58.4 122 27.6 12 2.7 
New Jersey 9 5.5 14 8.5 79 47.9 61 37.0 2 1.2 
New Mexico 1 2.3 1 2.3 29 67.4 8 18.6 4 9.3 
Ohio 5 5.5 16 17.6 45 49.5 17 18.7 8 8.8 
Rhode Island Hl 4.2 1 4.2 20 83.3 2 8.3 -- -- 
All States 65 4.5 105 7.3 781 54.4 426 29.7 59 4.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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PARCC Quality of Test Administration 


Test Administration Experience 


able H-8. TA Survey (PBA): Q1a. The policies and procedures within the Test Administrator Manual were easy 


‘or me to understand. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 5 5.7 13 14.8 47 53.4 23 26.1 
Colorado -- -- -- -- 13 81.3 3 18.8 
D.C. -- -- -- -- 2 100.0 -- -- 
Illinois 11 2.6 46 10.7 303 70.5 70 16.3 
Maryland 1 4.2 1 4.2 17 70.8 5 20.8 
Massachusetts 22 3.5 75 11.8 442 69.6 96 15.1 
Mississippi 1 2.5 1 2.5 25 62.5 13 32.5 
New Jersey 2 74 6 22.2 15 55.6 4 14.8 
New Mexico 2 3.6 7 12.5 41 73.2 6 10.7 
Ohio 14 6.3 32 14.4 155 69.8 21 9.5 
Rhode Island 7 5.3 18 13.6 32 69.7 15 11.4 
All States 109 4.8 272 12.1 1,513 67.2 358 15.9 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-9. TA Survey (EOY): Q1a. The policies and procedures within the Test Administrator Manual were easy 


for me to understand. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 12 6.5 18 9.7 127 68.3 29 15.6 
Colorado -- -- 1 9.1 8 72.7 2 18.2 
D.C. -- -- -- -- 2 100.0 -- -- 
Illinois 4 2.7 13 8.7 115 77.2 17 11.4 
Maryland 7 4.6 19 12.5 97 63.8 29 19.1 
Massachusetts 20 4.6 43 9.9 293 67.2 80 18.3 
New Jersey 6 3.6 18 10.7 116 68.6 29 17.2 
New Mexico -- -- 8 18.6 30 69.8 5 11.6 
Ohio 5 5.5 16 17.6 62 68.1 8 8.8 
Rhode Island -- -- 2 8.0 20 80.0 3 12.0 
All States 61 4.3 147 10.3 973 68.1 247 17.3 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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PARCC Quality of Test Administration 


able H-10. TA Survey (PBA): Qib. The instructions (including scripts for administering the assessment) within 


he Test Administrator Manual were easy for me to implement. 


Strongly Disagree Disagree Agree Strongly Agree 

State n % n % n % n % 
Arkansas 4 4.5 21 23.9 41 46.6 22 25.0 
Colorado 3 18.8 -- -- 10 62.5 3 18.8 
D.C. -- -- -- -- 2 100.0 -- -- 
Illinois 15 3.5 45 10.5 285 66.6 83 19.4 
Maryland 1 4.2 - - 15 62.5 8 33.3 
Massachusetts 17 2.7 49 7.8 443 70.1 123 19.5 
Mississippi 1 2.5 - - 25 62.5 14 35.0 
New Jersey -- -- 5 18.5 16 59.3 6 22.2 
New Mexico 2 3.6 10 17.9 41 73.2 3 5.4 
Ohio 14 6.3 22 9.9 156 70.3 30 13.5 
Rhode Island 8 6.1 9 6.9 92 70.2 22 16.8 
All States 113 5.0 232 10.4 1,473 65.7 423 18.9 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-11. TA Survey (EOY): Q1b. The instructions (including scripts for administering the assessment) within 


he Test Administrator Manual were easy for me to implement. 


Strongly Disagree Disagree Agree Strongly Agree 

State n % n % n % n % 
Arkansas 15 8.0 20 10.7 123 65.8 29 15.5 
Colorado -- -- -- -- 8 72.7 3 27.3 
D.C. -- -- -- -- 2 100.0 -- -- 
Illinois 4 24 20 13.4 105 70.5 20 13.4 
Maryland 7 4.6 24 15.9 86 57.0 34 22.5 
Massachusetts 16 3.7 30 6.9 288 66.5 99 22.9 
New Jersey 6 3.5 14 8.2 120 70.6 30 17.6 
New Mexico 1 2.3 5 11.6 31 72.1 6 14.0 
Ohio 2 2.2 11 12.0 68 73.9 11 12.0 
Rhode Island -- -- 4 16.0 16 64.0 5 20.0 
All States 58 4.1 142 9.9 941 65.9 287 20.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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PARCC Quality of Test Administration 


able H-12. TA Survey (PBA): Qic. Students appeared to understand the directions | read to them during test 


administration. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 11 12.6 10 11.5 48 55.2 18 20.7 
Colorado -- -- 3 18.8 10 62.5 3 18.8 
D.C. -- -- -- -- 2 100.0 -- -- 
Illinois 36 8.4 95 22.1 239 55.7 59 13.8 
Maryland 1 4.3 6 26.1 11 47.8 5 21.7 
Massachusetts 26 4.1 83 13.2 434 68.9 87 13.8 
Mississippi 3 7.7 4 10.3 21 53.8 11 28.2 
New Jersey -- -- 2 7.4 20 74.1 5 18.5 
New Mexico 7 12.7 9 16.4 37 67.3 2 3.6 
Ohio 24 10.8 37 16.7 140 63.1 21 9.5 
Rhode Island 6 4.5 24 18.2 88 66.7 14 10.6 
All States 161 7.2 372 16.6 1,400 62.6 304 13.6 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-13. TA Survey (EOY): Q1c. Students appeared to understand the directions | read to them during test 


administration. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 16 8.6 20 10.7 122 65.2 29 15.5 
Colorado -- -- 2 18.2 8 72.7 1 9.1 
D.C. -- -- -- -- 2 100.0 -- -- 
Illinois 5 3.4 32 21.6 92 62.2 19 12.8 
Maryland 13 8.8 19 12.8 88 59.5 28 18.9 
Massachusetts 17 3.9 47 10.9 299 69.1 70 16.2 
New Jersey 5 3.0 18 10.7 114 67.9 31 18.5 
New Mexico 2 4.7 9 20.9 25 58.1 7 16.3 
Ohio 4 4.4 12 13.2 64 70.3 11 12.1 
Rhode Island -- -- 4 16.0 17 68.0 4 16.0 
All States 69 4.9 176 12.4 937 65.9 239 16.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able H-14. TA Survey (PBA): Q1d. The instructions | read to the students covered all of the information 


necessary to take the test. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 9 10.3 14 16.1 44 50.6 20 23.0 
Colorado 2 12.5 2 12.5 10 62.5 2 12.5 
D.C. - -- -- -- 2 100.0 -- -- 
Illinois 29 6.8 84 19.7 258 60.6 55 12.9 
Maryland 1 4.2 4 16.7 15 62.5 4 16.7 
Massachusetts 25 4.0 136 21.7 384 61.2 82 13.1 
Mississippi 3 7.9 5 13.2 20 52.6 10 26.3 
New Jersey - -- 8 29.6 15 55.6 4 14.8 
New Mexico 7 12.5 11 19.6 34 60.7 4 7.1 
Ohio 20 9.0 42 18.8 141 63.2 20 9.0 
Rhode Island 6 4.6 29 22.3 83 63.8 12 9.2 
All States 160 7.2 457 20.5 1,330 59.6 284 12.7 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-15. TA Survey (EOY): Qid. The instructions | read to the students covered all of the information 


necessary to take the test. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arkansas 12 6.5 24 12.9 123 66.1 27 14.5 
Colorado 7 -- 2 18.2 9 81.8 -- -- 
D.C. _ -- -- -- 2 100.0 -- -- 
Illinois 8 5.4 28 18.8 98 65.8 15 10.1 
Maryland 13 8.6 35 23.2 80 53.0 23 15.2 
Massachusetts 18 4.2 70 16.2 276 63.9 68 15.7 
New Jersey 5 3.0 27 16.1 105 62.5 31 18.5 
New Mexico 2 4.7 6 14.0 28 65.1 7 16.3 
Ohio 3 3.3 19 20.9 60 65.9 9 9.9 
Rhode Island = -- 4 16.0 17 68.0 4 16.0 
All States 67 4.7 237 16.7 893 62.8 226 15.9 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able H-16. TA Survey (PBA): Qla-1d Summary Table 


State Mean Standard Deviation 
Arkansas 11.56 3.14 
Colorado 11.75 2.49 
D.C. 12.00 0.0 
Illinois 11.49 2.35 
Maryland 12.00 2.52 
Massachusetts 11.71 2.30 
Mississippi 12.33 2.93 
New Jersey 11.78 2.15 
New Mexico 10.91 2.29 
Ohio 11.13 2.46 
Rhode Island 11.31 2.52 
All States 11.46 2.53 


Note. a = .88. Scale scores were computed by taking the sum of items 1a-1d. 


able H-17. TA Survey (EOY): Qia-1d Summary Table 


State Mean Standard Deviation 
Arkansas 11.55 2.75 
Colorado 12.09 1.58 
D.C. 12.00 0.00 
Illinois 11.47 2.14 
Maryland 11.47 2.68 
Massachusetts 11.90 2.39 
New Jersey 11.91 2.27 
New Mexico 11.70 1.97 
Ohio 11.39 2.28 
Rhode Island 12.08 1.82 
All States 11.78 2.43 


Note. a = .89. Scale scores were computed by taking the sum of items 1a-1d. 
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Clarification on Mark Answers & Enter 
Instructions When to Use Calculator Responses After Completed Test Other 

State n % n % n % n % n % 
Arkansas 26 29.2 12 13.5 28 31.5 24 27.0 15 16.9 
Colorado 5 27.8 1 5.6 6 33.3 5 27.8 3 16.7 
D.C. -- -- -- -- 1 50.0 1 50.0 1 50.0 
Illinois 213 48.4 57 13.0 209 47.5 71 16.1 75 17.0 
Maryland 8 30.8 3 11.5 10 38.5 8 30.8 5 19.2 
Massachusetts 248 37.5 110 16.6 268 40.5 109 16.5 108 16.3 
Mississippi 7 17.5 3 7.5 13 32.5 2 5.0 8 20.0 
New Jersey 9 29.0 7 22.6 10 32.3 15 48.4 3 9.7 
New Mexico 19 33.3 12 21.1 25 43.9 22 38.6 7 12.3 
Ohio 102 42.7 36 15.1 111 46.4 61 25.5 47 19.7 
Rhode Island 51 37.5 20 14.7 39 28.7 19 14.0 22 16.2 
All States” 908 38.9 298 12.8 878 37.6 391 16.7 374 16.0 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
’N-counts include individuals with missing data for state location who selected a grade level. 
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Clarification on Mark Answers & Enter 

Instructions When to Use Calculator Responses After Completed Test Other 
State n % n % n % n % n % 
Arkansas 41 21.7 33 17.5 54 28.6 47 24.9 12 6.3 
Colorado 7 63.6 -- -- 6 54.5 1 9.1 -- -- 
D.C. 2 100.0 -- -- 1 50.0 1 50.0 -- -- 
Illinois 56 35.0 22 13.8 61 38.1 29 18.1 22 13.8 
Maryland 57 37.0 48 31.2 60 39.0 47 30.5 16 10.4 
Massachusetts 148 33.0 65 14.5 173 38.5 49 10.9 66 14.7 
New Jersey 40 23.1 36 20.8 57 32.9 57 32.9 20 11.6 
New Mexico 12 27.3 10 22.7 17 38.6 11 25.0 9 20.5 
Ohio 26 27.7 17 18.1 41 43.6 18 19.1 19 20.2 
Rhode Island 12 48.0 2 8.0 12 48.0 2 8.0 4 16.0 
All States” 438 29.8 240 16.3 510 34.7 276 18.8 184 12.5 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
>N-counts include individuals with missing data for state location who selected a grade level. 
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Very Early On Time Rush to Finish Did Not Finish 
State n % n % n % n % 
Arkansas 61 70.9 22 25.6 3 3.5 -- -- 
Colorado 9 56.3 7 43.8 -- -- -- -- 
D.C. 1 50.0 1 50.0 -- -- -- -- 
Illinois 154 36.5 252 59.7 12 2.8 4 0.9 
Maryland 7 30.4 15 65.2 -- -- 1 4.3 
Massachusetts 71 11.5 393 63.9 111 18.0 40 6.5 
Mississippi 3 7.9 24 63.2 9 23.7 2 5.3 
New Jersey 13 50.0 12 46.2 1 3.8 -- -- 
New Mexico 9 16.7 37 68.5 7 13.0 1 1.9 
Ohio 61 27.9 139 63.5 16 7.3 3 1.4 
Rhode Island 48 36.9 73 56.2 5 3.8 4 3.1 
All States 556 25.5 1,351 61.9 215 9.8 61 2.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-21. TA Survey (EOY): Q3. Did your students have sufficient time to finish the test? 


Very Early On Time Rush to Finish Did Not Finish 
State n % n % n % n % 
Arkansas 106 57.6 68 37.0 5 2.7 5 2.7 
Colorado 5 45.5 6 54.5 -- -- -- -- 
D.C. 1 50.0 1 50.0 -- -- -- - 
Illinois 67 45.0 76 51.0 5 3.4 1 0.7 
Maryland 102 67.1 49 32.2 1 0.7 -- -- 
Massachusetts 75 17.9 264 63.2 56 13.4 23 5.5 
New Jersey 93 57.4 66 40.7 1 0.6 2 1.2 
New Mexico 7 16.3 28 65.1 8 18.6 -- -- 
Ohio 27 30.7 55 62.5 6 6.8 -- -- 
Rhode Island 6 25.0 16 66.7 1 4.2 1 4.2 
All States 522 37.6 726 52.2 105 7.6 37 2.7 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Accessibility Features and Accommodations 


able H-22. TA Survey (PBA): Q1. Did you read the “PARCC Accessibility Features and Accommodations Manual” 


prior to administration? 


Yes No 

State n % n % 
Arkansas 74 84.1 14 15.9 
Colorado 9 56.3 7 43.8 
D.C. 2 100.0 -- -- 
Illinois 330 77.6 95 22.4 
Maryland 20 83.3 4 16.7 
Massachusetts 512 82.1 112 17.9 
Mississippi 18 48.6 19 51.4 
New Jersey 22 88.0 3 12.0 
New Mexico 44 78.6 12 21.4 
Ohio 188 85.5 32 14.5 
Rhode Island 113 86.9 17 13.1 
All States 1,872 84.0 356 16.0 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-23. TA Survey (EOY): Q1. Did you read the “PARCC Accessibility Features and Accommodations Manual” 


prior to administration? 


Yes No 

State n % n % 
Arkansas 143 77.3 42 22.7 
Colorado 8 72.7 3 27.3 
D.C. 2 100.0 -- -- 
Illinois 116 77.3 34 22.3 
Maryland 118 78.1 33 21.9 
Massachusetts 372 87.1 55 12.9 
New Jersey 136 81.0 32 19.0 
New Mexico 36 85.7 6 14.3 
Ohio 70 77.8 20 22.2 
Rhode Island 19 76.0 6 24.0 
All States 1,172 85.9 242 17.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able H-24. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


PARCC Accessibility Features & Accommodations Manual—3" Edition 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 2 2.4 6 7.1 46 54.8 11 13.1 19 22.6 
Colorado -- -- 1 7.1 6 42.9 2 14.3 5 35.7 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 11 2f 41 10.1 195 48.3 47 11.6 110 27.2 
Maryland 1 4.3 2 8.7 11 47.8 5 21.7 4 17.4 
Massachusetts 12 2.1 76 13.0 323 55.3 46 7.9 127 21.7 
Mississippi == - 2 5.1 16 41.0 8 20.5 13 33.3 
New Jersey -- - 4 16.0 13 52.0 4 16.0 4 16.0 
New Mexico 3 5.8 4 7.7 26 50.0 5 9.6 14 26.9 
Ohio 16 7.7 33 15.9 102 49.3 10 4.8 46 22.2 
Rhode Island 5 4.1 15 12.2 71 57.7 6 4.9 26 21.1 
All States 68 3.2 220 10.5 1,145 54.4 207 9.8 465 22.1 
Accessibility & Accommodations Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 4 4.8 8 9.5 42 50.0 12 14.3 18 21.4 
Colorado -- -- 1 7.1 4 28.6 2 14.3 7 50.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 15 3.8 47 11.8 147 36.8 32 8.0 158 39.6 
Maryland 1 4.3 2 8.7 7 30.4 4 17.4 9 39.1 
Massachusetts 17 3.0 72 12.5 257 44.7 31 5.4 198 34.4 
Mississippi -- - 4 10.5 13 34.2 7 18.4 14 36.8 
New Jersey -- -- 4 16.7 13 54.2 1 4.2 6 25.0 
New Mexico 3 5.8 3 5.8 26 50.0 5 9.6 15 28.8 
Ohio 19 9.3 31 15.2 76 37.3 6 2.9 72 35.3 
Rhode Island 4 3.3 15 12.2 45 36.6 4 3.3 55 44.7 
All States 75 3.6 224 10.8 893 42.9 152 7.3 736 35.4 
Personal Needs Profile Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 2 2.5 7 8.6 40 49.4 9 11.1 23 28.4 
Colorado -- -- 1 7.7 3 23.1 1 7.7 8 61.5 
D.C. -- -- -- -- 1 50.0 1 50.0 -- -- 
Illinois 15 3.8 49 12.4 96 24.4 20 5.1 214 54.3 
Maryland -- -- 1 4.3 6 26.1 4 17.4 12 52.2 
Massachusetts 15 2.6 66 11.6 186 32.6 25 4.4 278 48.8 
Mississippi - - 4 10.3 12 30.8 6 15.4 17 43.6 
New Jersey -- -- 6 24.0 9 36.0 2 8.0 8 32.0 
New Mexico 3 5.8 2 3.8 22 42.3 4 7.7 21 40.4 
Ohio 16 7.9 34 16.7 57 28.1 3 15 93 45.8 
Rhode Island 7 5.7 15 12.3 30 24.6 4 3.3 66 54.1 
All States 71 3.5 219 10.7 673 32.8 113 5.5 976 47.6 

(continued) 
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able H-24. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


Personal Needs Profile Field Definitions 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 4 4.9 7 8.6 40 49.4 9 11.1 21 25.9 
Colorado -- -- 1 7.7 3 23.1 al 7.7 8 44.4 
D.C. -- -- -- -- 1 50.0 1 50.0 -- -- 
Illinois 18 4.5 48 12.1 102 25.6 21 5.3 209 52.5 
Maryland -- -- -- -- 8 36.4 3 13.6 11 50.0 
Massachusetts 14 2.5 67 11.8 198 34.7 26 4.6 265 46.5 
Mississippi -- - 5 13.2 11 28.9 6 15.8 16 42.1 
New Jersey -- -- 7 28.0 6 24.0 2 8.0 10 40.0 
New Mexico 3 6.3 2 4.2 22 45.8 2 4.2 19 39.6 
Ohio 16 8.0 32 16.0 61 30.5 5 2.5 86 43.0 
Rhode Island 5 4.3 14 12.0 34 29.1 3 2.6 61 52.1 
All States 72 3.5 223 10.9 711 34.7 116 5.7 927 45.2 
Training or resources from my state department of education 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 5 6.2 9 11.1 36 44.4 10 12.3 21 25.9 
Colorado 1 7.7 1 7.7 3 23.1 1 7.7 7 53.8 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 34 8.6 56 14.2 82 20.8 24 6.1 198 50.3 
Maryland -- -- -- -- 9 40.9 1 4.5 12 54.5 
Massachusetts 39 6.8 85 14.9 156 27.3 28 4.9 264 46.2 
Mississippi 3 7.9 4 10.5 10 26.3 6 15.8 15 39.5 
New Jersey 2 8.3 7 29.2 7 29.2 al 4.2 7 29.2 
New Mexico 5 9.8 6 11.8 25 49.0 1 2.0 14 27.5 
Ohio 31 15.3 35 17.3 52 25.7 4 2.0 80 39.6 
Rhode Island 7 5.8 22 18.2 30 24.8 2 1.7 60 49.6 
All States 163 7.9 281 13.7 620 30.1 117 5.7 877 42.6 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 6 7.1 5 6.0 47 56.0 14 16.7 12 14.3 
Colorado 1 7.7 2 15.4 3 23.1 1 7.7 6 46.2 
D.C. -- -- -- -- 1 50.0 -- -- 1 50.0 
Illinois 20 5.0 32 8.0 155 38.8 59 14.8 134 33.5 
Maryland -- -- 2 8.7 8 34.8 5 21.7 8 34.8 
Massachusetts 29 5.0 79 13.7 225 39.1 57 9.9 186 32.3 
Mississippi 2 5.1 2 5.1 16 41.0 12 30.8 7 17.9 
New Jersey - - 4 16.0 14 56.0 4 16.0 3 12.0 
New Mexico 4 8.0 4 8.0 23 46.0 5 10.0 14 28.0 
Ohio 17 8.3 21 10.2 95 46.1 23 11.2 50 24.3 
Rhode Island 6 4.8 21 16.8 51 40.8 5 3.7 42 30.9 
All States 111 5.3 217 10.4 862 41.2 276 13.2 625 29.9 


(continued) 
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able H-24. TA Survey (PBA): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 
Training from my school 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 3 3.6 3 3.6 52 62.7 18 21.7 7 8.4 
Colorado -- -- 2 14.3 9 64.3 3 21.4 -- -- 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 13 3.3 18 4.5 201 50.5 85 21.4 81 20.4 
Maryland -- -- 2 8.7 10 43.5 9 39.1 2 8.7 
Massachusetts 21 3.5 51 8.6 326 55.1 133 22.5 61 10.3 
Mississippi 2 5.1 4 10.3 13 33.3 14 35.9 6 15.7 
New Jersey -- - 3 12.0 14 56.0 6 24.0 2 8.0 
New Mexico 5 9.6 28 53.8 13 25.0 6 11.5 
Ohio 16 7.7 20 9.7 108 52.2 30 14.5 33 15.9 
Rhode Island 5 4.0 17 13.5 66 52.4 11 8.7 27 21.4 
All States 84 4.0 144 6.8 1,142 53.7 496 23.3 259 12.2 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able H-25. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


PARCC Accessibility Features & Accommodations Manual—3" Edition 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 7 3.9 15 8.4 112 62.6 18 10.1 27 15.1 
Colorado -- -- 1 10.0 7 70.0 1 10.0 a 10.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 1 0.7 14 10.1 80 57.6 9 6.5 35 25.2 
Maryland 7 4.9 16 11.3 77 54.2 11 7.7 31 21.8 
Massachusetts 14 3.6 41 10.4 226 57.4 44 11.2 69 17.5 
New Jersey 5 3.3 8 5.3 92 61.3 8 5.3 37 24.7 
New Mexico 1 2.5 4 10.0 26 65.0 2 5.0 7 17.5 
Ohio 1 1.2 9 10.7 41 48.8 5 6.0 28 33.3 
Rhode Island -- -- 2 8.7 13 56.5 1 4.3 7 30.4 
All States 42 3.2 115 8.7 768 58.2 132 10.0 262 19.9 
Accessibility & Accommodations Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 5.1 14 7.9 103 57.9 15 8.4 37 20.8 
Colorado -- -- 1 10.0 8 80.0 -- -- 41 10.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 3 2.2 16 11.9 56 41.5 6 4.4 54 40.0 
Maryland 8 5.6 16 11.2 56 39.2 8 5.6 55 38.5 
Massachusetts 15 3.8 48 12.2 160 40.8 27 6.9 142 36.2 
New Jersey 5 3.4 11 7.6 74 54.0 2 1.4 53 36.6 
New Mexico al 2.6 4 10.3 23 59.0 2 5.1 9 23.1 
Ohio 2 2.4 9 10.8 28 33.7 3 3.6 41 49.4 
Rhode Island 1 4.5 1 4.5 8 36.4 1 4.5 11 50.0 
All States 51 3.9 126 9.7 604 46.3 88 6.7 435 33.4 
Personal Needs Profile Training Module 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 5.0 12 6.7 93 52.0 13 7.3 52 29.1 
Colorado -- -- il 10.0 6 60.0 -- -- 3 30.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 2 1.5 15 11.0 47 34.6 3 2.2 69 50.7 
Maryland 7 4.9 13 9.2 38 26.8 5 3.5 79 55.6 
Massachusetts 17 4.4 45 11.6 115 29.6 14 3.6 197 50.8 
New Jersey 5 3.4 14 9.6 56 38.4 2 1.4 69 47.3 
New Mexico -- -- 7 17.9 19 48.7 1 2.6 12 30.8 
Ohio 1 1.2 8 9.6 28 33.7 ul 1.2 45 54.2 
Rhode Island 2 9.1 1 4.5 5 22.7 1 4.5 13 59.1 
All States 47 3.6 124 9.5 488 37.5 57 4.4 584 44.9 


(continued) 
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able H-25. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 


Personal Needs Profile Field Definitions 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 5.1 14 7.9 88 49.7 13 7.3 53 29.9 
Colorado -- -- -- -- 6 60.0 -- -- 4 40.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 2 1.5 17 12.6 47 34.8 3 2:2 66 48.9 
Maryland 7 5.0 13 9.2 34 24.1 6 4.3 81 57.4 
Massachusetts 19 4.9 46 11.9 120 31.1 13 3.4 188 48.7 
New Jersey 5 3.5 13 9.1 53 37.1 4 2.8 68 47.6 
New Mexico -- -- 8 21.1 18 47.4 -- -- 12 31.6 
Ohio 3 3.6 7 8.4 35 42.2 3 3.6 35 42.2 
Rhode Island 1 5.0 2 10.0 4 20.0 1 5.0 12 60.0 
All States 49 3.8 131 10.2 488 37.8 60 4.7 562 43.6 
Training or resources from my state department of education 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 8 4.5 17 9.6 94 52.8 11 6.2 48 27.0 
Colorado -- -- -- -- 4 40.0 -- -- 6 60.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 12 8.9 23 17.0 38 28.1 5 3.7 57 42.2 
Maryland 8 5.6 26 18.2 32 22.4 4 2.8 73 51.0 
Massachusetts 22 5.6 71 18.2 102 26.2 15 3.8 180 46.2 
New Jersey 12 8.3 26 17.9 51 35.2 3 2.1 53 36.6 
New Mexico al 2.6 10 25.6 16 1.0 1 2.6 11 28.2 
Ohio 7 8.4 9 10.8 24 28.9 3 3.6 40 48.2 
Rhode Island 3 13.6 -- -- 3 13.6 -- -- 16 72.7 
All States 80 6.1 197 15.1 436 33.5 59 4.5 529 40.7 
Training from my district 

Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 7 3.9 14 7.8 106 58.9 28 15.6 25 13.9 
Colorado -- -- -- -- 5 50.0 2 20.0 3 30.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 6 4.4 19 13.9 56 40.9 14 10.2 42 30.7 
Maryland 10 7.0 21 14.8 51 35.9 5 3.5 55 38.7 
Massachusetts 16 4.1 47 12.1 151 38.7 57 14.6 119 30.5 
New Jersey 10 6.6 12 7.9 77 51.0 30 19.9 22 14.6 
New Mexico 2 5.0 5 12.5 24 60.0 1 2.5 8 20.0 
Ohio 5 5.8 10 11.6 34 39.5 11 12.8 26 30.2 
Rhode Island 1 4.5 1 4.5 7 31.8 -- -- 13 59.1 
All States 61 4.6 141 10.7 591 44.9 180 13.7 344 26.1 


(continued) 
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able H-25. TA Survey (EOY): Q2. Please indicate your level of agreement with the following statement: “This 


particular item/component effectively prepared me to administer accessibility features and accommodations on 
he PARCC assessments.” 
Training from my school 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 8 4.4 10 5.6 114 63.3 40 22.2 8 4.4 
Colorado -- -- -- -- 5 50.0 5 50.0 -- -- 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 4 2.9 15 10.9 71 51.8 21 15.3 26 19.0 
Maryland 6 4.2 5 3.5 78 54.9 41 28.9 12 8.5 
Massachusetts 12 3.0 35 8.7 225 55.8 102 25.3 29 7.2 
New Jersey 6 3.9 15 9.9 78 51.3 39 25.7 14 9.2 
New Mexico 1 2.6 3 7.7 29 74.4 1 2.6 5 12.8 
Ohio 3 3.5 14 16.5 32 37.6 14 16.5 22 25.9 
Rhode Island 2 9.1 1 4.5 14 63.6 1 4.5 4 18.2 
All States 45 3.4 102 7.7 730 54.8 321 24.1 133 10.0 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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PARCC Quality of Test Administration 


able H-26. TA Survey (PBA): Q3. Did you administer the following accessibility features or accommodations to any of your students? Select all that apply.* 


Test Calculator 
Human Human Directions Word on Non- 
Reader Reader Human Human Native Text-to- Extended Prediction Calculator Other Tech 
Math English Scribe Signer Language Speech Time Device Section Device 

State n % n % n % n % n % n % n % n % n % n % 
Arkansas 24 27.0 8 9.0 5 5.6 -- -- 5 5.6 1 1.1 31 34.8 1 1.1 11 «612.4 5 5.6 
Colorado 2 11.1 -- -- -- -- -- -- 2 11.1 -- -- 4 22.2 -- -- -- -- 1 5.6 
Illinois 102 23.2 44 10.0 43 9.8 2 0.5 24 5.5 3 0.7 142 32.3 1 0.2 60 13.6 5 1.1 
Maryland 6 23.1 3 11.5 3 11.5 -- -- -- -- -- -- 10 38.5 -- -- 7 26.9 -- -- 
Massachusetts 184 27.8 102 15.4 107 16.2 4 0.6 42 6.3 5 0.8 294 44.4 4 0.6 79 =11.9 37 5.6 
Mississippi 6 15.0 4 10.0 -- -- -- -- 3 7.5 -- -- 6 15.0 -- -- 1 2.5 -- -- 
New Jersey 5 16.1 7 22.6 3 9.7 - -- 2 6.5 3 9.7 17 54.8 -- -- 15 48.4 1 3.2 
New Mexico 13 22.8 9 15.8 4 7.0 - -- 4 7.0 1 1.8 16 28.1 -- - 5 8.8 2 3.5 
Ohio 84 35.1 39 «16.3 24 10.0 2 0.8 16 6.7 al 0.4 89 37.2 -- -- 25 10.5 a 0.4 
Rhode Island 15 11.0 6 4.4 5 3.7 1 0.7 4 2.9 - -- 330 24.3 1 0.7 4 2.9 2 1.5 
All States” 630 27.0 356 15.2 236 10.1 11 0.5 133 5.7 23 1.0 892 38.2 9 0.4 355 15.2 70 3.0 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
’N-counts include individuals with missing data for state location who selected a grade level. 
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PARCC Quality of Test Administration 


able H-27. TA Survey (EOY): Q3. Did you administer the following accessibility features or accommodations to any of your students? Select all that apply.* 


Test Calculator 
Human Human Directions Word on Non- 
Reader Reader Human Human Native Text-to- Extended Prediction Calculator Other Tech 
Math English Scribe Signer Language Speech Time Device Section Device 

State n % n % n % n % n % n % n % n % n % n % 
Arkansas 44 23.3 29 «15.3 6 3.2 -- -- 12 6.3 2 1.1 34 18.0 -- -- 8 4.2 4 2.1 
Colorado 1 9.1 1 9.1 1 9.1 -- -- -- -- -- -- 2 18.2 -- -- -- -- -- -- 
Illinois 28 17.5 11 6.9 5 3.1 -- -- 9 5.6 -- -- 39 244 -- -- 12 7.5 -- -- 
Maryland 26 16.9 21 = 13.6 28 18.2 -- -- 1 0.6 -- -- 51 33.1 -- -- 41 26.6 5 3.2 
Massachusetts 141 31.4 75 16.7 80 17.8 3 0.7 39 8.7 1 0.2 211 47.0 5 1.1 53 11.8 17 3.8 
New Jersey 22 «12.7 21 «12.1 11 6.4 a 0.6 5 2.9 7 4.0 59 34.1 1 0.6 46 26.6 5 2.9 
New Mexico 9 20.5 2 4.5 2 4.5 -- -- 2 4.5 -- -- 14 31.8 -- -- 4 9.1 -- -- 
Ohio 27 =. 28.7 13 13.8 9 9.6 -- -- 6 6.4 -- -- 36 = 338.3 -- -- 14 149 -- -- 
Rhode Island 3 12.0 2 8.0 a 4.0 -- -- 2 8.0 -- -- 7 = 28.0 -- -- -- -- -- -- 
All States” 366 24.9 229 15.6 157 10.7 5 0.3 81 5.5 12 0.8 538 36.6 10 0.7 242 16.5 36 2.5 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “ 
selected” responses. 
>N-counts include individuals with missing data for state location who selected a grade level. 
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Quality of Test Administration 


able H-28. TA Survey (PBA): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 


Appendix A: Accessibility Features & Accommodations for Students taking the Paper-Based PARCC Assessments 


Strongly Disagree Disagree Agree Strongly Agree —_ Did Not Administer 

State n % n % n % n % n % 

Arkansas 2 2.5 3 3.7 38 46.9 9 11.1 29 35.8 
Colorado -- -- -- -- 3 27.3 1 9.1 7 63.6 
D.C. -- -- -- -- -- -- 1 50.0 1 50.0 
Illinois 7 1.8 16 4.1 156 39.6 30 7.6 185 47.0 
Maryland 1 4.2 -- -- 11 45.8 2 8.3 10 41.7 
Massachusetts 13 2.9 45 8.3 269 49.4 41 7.5 173 31.8 
Mississippi - -- 1 2.9 8 22.9 8 22.9 18 51.4 
New Jersey - -- 2 8.0 14 56.0 2 8.0 7 28.0 
New Mexico 2 4.1 4 8.2 17 34.7 4 8.2 22 44.9 
Ohio 11 5.5 24 12.1 90 45.2 5 2.5 69 34.7 
Rhode Island 2 1.7 9 7.6 53 44.5 7 5.9 48 40.3 
All States 50 2.5 128 6.4 926 46.0 168 8.4 739 36.7 


Appendix B: Test Administration Protocol for the Human Reader Accommodations for ELA/L and the Human 


Reader Accessibility Feature for Math 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 3 3.8 2 2.5 32 40.0 5 6.3 38 47.8 
Colorado -- -- -- -- 2 18.2 1 9.1 8 72.7 
D.C. -- -- -- -- 1 50.0 -- -- 1 50.0 
Illinois 12 3.1 16 4.1 97 24.8 18 4.6 248 63.4 
Maryland 1 4.2 2 8.3 3 12.5 4 16.7 14 58.3 
Massachusetts 12 2.2 29 5.4 169 31.2 26 4.8 305 56.4 
Mississippi -- -- 1 2.9 7 20.0 6 17.1 21 60.0 
New Jersey -- - 3 12.0 8 32.0 1 4.0 13 52.0 
New Mexico 2 4.0 4 8.0 12 24.0 4 8.0 28 56.0 
Ohio 9 4.6 22 11.2 66 33.5 4 2.0 96 48.7 
Rhode Island 4 3.4 7 5.9 19 16.1 4 3.4 84 71.2 
All States 50 2.5 107 5.3 612 30.5 119 5.9 1,116 55.7 


Appendix C: Protocol for the Use of the Scribe and for Transcribing Student Responses 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 1 1.3 2 2.5 26 32.5 7 8.8 44 55.0 
Colorado -- -- -- -- 2 18.2 1 9.1 8 72.7 
D.C. -- -- -- -- -- -- -- -- 2 100.0 
Illinois 7 1.8 15 3.9 75 19.3 13 3.4 278 71.6 
Maryland -- -- 1 4.3 7 30.4 2 8.7 13 56.5 
Massachusetts 13 2.4 35 6.5 134 24.8 14 2.6 345 63.8 
Mississippi -- - 1 2.9 5 14.3 5 14.3 24 68.6 
New Jersey -- -- 3 12.0 6 24.0 -- -- 16 64.0 
New Mexico 2 4.2 2 4.2 10 20.8 4 8.3 30 62.5 
Ohio 7 3.6 18 9.2 47 24.0 3 15 121 61.7 
Rhode Island 2 1.7 4 3.4 16 13.8 2 1.7 92 79.3 
All States 37 1.9 100 5.0 463 23.3 75 3.8 1,311 66.0 
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PARCC Quality of Test Administration 


able H-28. TA Survey (PBA): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Appendix E: Guidance for Selecting and Administering the Extended Time Accommodation 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 1 1.3 2 2.5 34 43.0 9 11.4 33 41.8 
Colorado -- -- 1 9.1 4 36.4 1 9.1 5 45.5 
D.C. -- -- -- -- -- -- -- -- 2 100.0 
Illinois 8 2.1 12 3.1 122 31.3 22 5.6 226 57.9 
Maryland -- -- -- -- 10 41.7 4 16.7 10 41.7 
Massachusetts 13 2.4 33 6.1 229 42.3 31 5.7 236 43.5 
Mississippi -- = 1 2.9 7 20.0 7 20.0 20 57.1 
New Jersey -- -- -- -- 14 58.3 2 8.3 8 33.3 
New Mexico i 2.2 2 4.3 13 28.3 7 15.2 23 50.0 
Ohio 7 3.6 19 9.7 72 36.7 5 2.6 93 47.4 
Rhode Island 2 1.7 3 2.5 31 26.3 9 7.6 73 61.9 
All States 39 2.0 91 4.6 767 38.5 148 7.4 949 47.6 
Appendix I: PARCC ELA Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 2 2.5 1 1.3 26 32.5 6 75 45 56.3 
Colorado -- -- -- -- 3 27.3 1 9.1 7 63.6 
D.C. -- -- -- -- -- -- -- -- 2 100.0 
Illinois 6 1.6 11 2.8 46 11.9 10 2.6 313 81.1 
Maryland -- -- 1 4.3 3 13.0 2 8.7 17 73.9 
Massachusetts 7 1.3 13 2.4 61 11.3 11 2.0 447 82.9 
Mississippi -- -- 1 2.9 5 14.3 5 14.3 24 68.8 
New Jersey -- -- 2 8.0 7 28.0 -- -- 16 64.0 
New Mexico 2 4.2 3 6.6 8 16.7 4 8.3 31 64.6 
Ohio 9 4.6 15 7.7 30 15.5 1 0.5 139 71.6 
Rhode Island 2 1.7 3 2.5 8 6.8 1 0.8 104 88.1 
All States 33 1.7 68 3.4 310 15.7 58 2.9 1,507 76.3 
Appendix J: PARCC Math Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 2 2.5 2 2.5 26 32.9 6 7.6 43 54.4 
Colorado -- -- -- -- 2 18.2 1 9.1 8 72.7 
D.C. -- -- -- -- -- -- -- -- 2 100.0 
Illinois 6 1.6 12 3.1 45 11.7 11 2.9 311 80.8 
Maryland -- -- 1 4.2 3 12.5 2 8.3 18 75.0 
Massachusetts 8 1.5 13 2.4 62 11.5 10 1.9 444 82.7 
Mississippi -- -- 1 2.9 5 14.3 5 14.3 24 68.6 
New Jersey -- -- 3 13.0 6 26.1 -- - 14 60.9 
New Mexico 1 2.1 3 6.4 10 21.3 4 8.5 29 61.7 
Ohio 9 4.6 13 6.6 35 17.8 1 0.5 139 70.6 
Rhode Island 2 1.7 2 1.7 10 8.5 al 0.9 102 87.2 
All States 33 1.7 69 3.5 328 16.6 59 3.0 1,484 75.2 


(continued) 
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PARCC Quality of Test Administration 


able H-28. TA Survey (PBA): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Assistive Technology Guidelines 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 1 1.3 3 3.8 23 28.8 5 6.3 48 60.0 
Colorado -- -- -- -- 2 18.2 1 9.1 8 72.7 
D.C. -- -- -- -- -- -- -- -- 2 100.0 
Illinois 6 1.6 14 3.6 48 12.5 10 2.6 307 79.7 
Maryland -- -- -- -- 5 20.8 2 8.3 17 70.8 
Massachusetts 14 2.6 21 3.9 77 14.4 9 1.7 414 774 
Mississippi -- - 1 2.9 5 14.7 5 14.7 23 67.6 
New Jersey -- -- 3 12.0 4 16.0 1 4.0 17 68.0 
New Mexico 2 4.3 3 6.4 6 12.8 3 6.4 33 70.2 
Ohio 8 4.1 15 7.8 27 14.0 1 0.5 142 73.6 
Rhode Island 3 2.6 3 2.6 11 9.4 1 0.9 99 84.6 
All States 38 1.9 86 4.4 353 18.0 63 3.2 1,426 72.5 


PARCC Technical Assistance Bulletin - PARCC Assessments and Students with Visual Impairment, Including 
Blindness 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 1 1.3 2 2.6 24 31.2 6 7.8 44 57.1 
Colorado -- -- -- -- 2 18.2 1 9.1 8 72.7 
D.C. -- -- -- -- -- -- -- -- 2 100.0 
Illinois 5 1.3 11 2.9 41 10.7 -- -- 319 83.3 
Maryland -- -- -- -- 5 20.8 -- -- 19 79.2 
Massachusetts 8 1.5 11 2.0 61 11.3 7 1.3 453 83.9 
Mississippi -- -- 1 2.9 5 14.3 5 14.3 24 68.6 
New Jersey -- -- 2 8.0 4 16.0 -- -- 19 76.0 
New Mexico 2 4.3 2 4.3 8 17.0 3 6.4 32 68.1 
Ohio 7 3.6 16 8.2 24 12.4 -- -- 24 12.4 
Rhode Island 3 2.6 3 2.6 8 6.9 1 0.9 101 87.1 
All States 29 15 63 3.2 283 14.4 44 2.2 1,547 78.7 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Quality of Test Administration 


able H-29. TA Survey (EOY): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 


Appendix A: Accessibility Features & Accommodations for Students taking the Paper-Based PARCC Assessments 


Strongly Disagree Disagree Agree Strongly Agree — Did Not Administer 

State n % n % n % n % n % 

Arkansas 5 2.9 7 4.0 77 44.0 14 8.0 72 41.1 
Colorado -- -- 1 10.0 2 20.0 1 10.0 6 60.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 

Illinois 1 0.8 7 5.3 50 38.2 5 3.8 68 51.9 
Maryland 5 3.7 3 2.2 64 47.8 9 6.7 53 39.6 
Massachusetts 12 3.2 33 8.7 185 48.7 43 11.3 107 28.2 
New Jersey 3 2.1 3 2.1 37 25.5 5 3.4 97 66.9 
New Mexico 1 2.7 2 5.4 17 45.9 1 2.7 16 43.2 
Ohio 2 2.4 5 6.1 35 42.7 2 2.4 38 46.3 
Rhode Island -- -- -- -- 11 47.8 2 8.7 10 43.5 
All States 32 2.5 70 5.5 560 44.1 112 8.8 497 39.1 


Appendix B: Test Administration Protocol for the Human Reader Accommodations for ELA/L and the Human 


Reader Accessibility Feature for Math 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 6 3.6 8 4.7 60 35.5 7 4.1 88 52.1 
Colorado -- - -- - 1 10.0 -- - 9 90.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 

Illinois 1 0.8 6 4.6 30 23.1 3 2.3 90 69.2 
Maryland 2 1.5 5 3.8 30 22.6 7 5.3 89 66.9 
Massachusetts 10 2.6 23 6.1 121 31.9 28 74 197 52.0 
New Jersey 3 2.1 4 2.8 30 21.0 5 3.5 101 70.6 
New Mexico 1 2.7 3 8.1 13 35.1 1 2.7 19 51.4 
Ohio 4 4.9 4 4.9 23 28.0 -- -- 51 62.2 
Rhode Island -- -- 1 4.8 -- -- 1 4.8 19 90.5 
All States 30 2.4 59 4.7 374 29.9 72 5.8 717 57.3 


Appendix C: Protocol for the Use of the Scribe and for Transcribing Student Responses 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 

State n % n % n % n % n % 

Arkansas 4 2.4 8 4.8 47 28.1 6 3.6 102 61.1 
Colorado -- -- -- -- 2 20.0 -- -- 8 80.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 

Illinois 2 1.5 5 3.8 16 12.3 -- -- 107 82.3 
Maryland 2 1.5 5 3.8 34 25.8 7 5.3 84 63.6 
Massachusetts 9 2.4 21 5.6 91 24.3 20 5.3 233 62.3 
New Jersey 3 2.1 5 3.4 31 21.2 5 3.4 102 69.9 
New Mexico 1 2.7 2 5.4 10 27.0 -- -- 24 64.9 
Ohio 3 3.7 3 3.7 20 24.4 -- -- 56 68.3 
Rhode Island -- -- -- -- -- -- 1 4.8 20 95.2 
All States 27 2.2 52 4.2 302 24.3 52 4.2 812 65.2 
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PARCC Quality of Test Administration 


able H-29. TA Survey (EOY): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Appendix E: Guidance for Selecting and Administering the Extended Time Accommodation 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 4 2.4 7 4.1 63 37.3 9 5.3 86 50.9 
Colorado 1 10.0 -- -- 2 20.0 -- -- 7 70.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 1 0.8 6 4.6 37 28.2 4 3.1 83 63.4 
Maryland 3 2.3 5 3.8 45 34.1 8 6.1 71 53.8 
Massachusetts 8 2.1 24 6.3 149 39.3 42 11.1 156 41.2 
New Jersey 3 2.1 7 4.8 48 32.9 8 5.5 80 54.8 
New Mexico -- -- 1 2.7 16 43.2 1 2.7 19 51.4 
Ohio 3 3.8 2 2.5 34 42.5 1 1.3 40 50.0 
Rhode Island -- -- -- -- 5 23.8 1 4.8 15 71.4 
All States 25 2.0 54 4.3 474 37.8 97 7.7 603 48.1 
Appendix I: PARCC ELA Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 5 3.0 7 4.2 45 26.9 4 2.4 106 63.5 
Colorado -- -- id 10.0 4 10.0 -- -- 8 80.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 1 0.8 4 3.1 11 8.5 1 0.8 112 86.8 
Maryland 2 1.5 -- -- 14 10.6 5 3.8 111 84.1 
Massachusetts 5 1.3 13 3.5 41 11.0 11 2.9 304 81.3 
New Jersey 3 2.1 6 4.1 31 21.2 6 4.1 100 68.5 
New Mexico -- - 1 2.8 10 27.8 -- - 25 69.4 
Ohio 1. 1.2 1 1.2 13 15.9 -- -- 67 81.7 
Rhode Island -- -- -- -- -- -- -- -- 21 100.0 
All States 20 1.6 34 2.7 216 17.4 36 2.9 937 75.4 
Appendix J: PARCC Math Audio Guidelines 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 5 3.0 7 4.2 48 28.7 4 2.4 103 61.7 
Colorado -- -- 1 10.0 1 10.0 -- -- 8 80.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 1 0.8 4 3.1 12 9.3 1 0.8 111 86.0 
Maryland 2 1.5 -- -- 14 10.6 4 3.0 112 84.8 
Massachusetts 7 1.9 12 3.2 47 12.5 11 2.9 298 79.5 
New Jersey 3 2.1 5 3.4 30 20.5 6 4.1 102 69.9 
New Mexico -- = 1 2.7 14 37.8 - - 22 59.5 
Ohio 1 1.2 1 1.2 13 16.0 -- -- 66 81.5 
Rhode Island -- -- -- -- -- -- -- -- 21 100.0 
All States 22 1.8 32 2.6 231 18.6 37 2.7 925 74.4 


(continued) 
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Quality of Test Administration 


able H-29. TA Survey (EOY): Q4. Please indicate your level of agreement with the following statement: “This 


particular appendix/guidance effectively informed me of how to administer accessibility features and 
laccommodations on the PARCC assessments.” 
Assistive Technology Guidelines 


Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 4 2.4 9 5.4 42 25.3 6 3.6 105 63.3 
Colorado -- -- -- -- 4. 10.0 -- -- 9 10.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 1 0.8 4 3.1 13 10.1 1 0.8 110 85.3 
Maryland 4 3.1 1 0.8 19 14.5 5 3.8 102 77.9 
Massachusetts 4 1.1 21 5.6 45 12.1 12 3.2 290 78.0 
New Jersey 3 2.1 5 3.4 20 13.7 6 4.1 112 76.7 
New Mexico -- -- 1 2.8 10 27.8 -- -- 25 69.4 
Ohio 2 2.4 dl. 1.2 13 15.9 -- -- 66 80.5 
Rhode Island -- -- -- -- -- -- -- -- 21 100.0 
All States 22 1.8 46 3.7 222 17.9 42 3.4 909 73.2 
PARCC Technical Assistance Bulletin - PARCC Assessments and Students with Visual Impairment, Including 
Blindness 

Strongly Disagree Disagree Agree Strongly Agree Did Not Administer 
State n % n % n % n % n % 
Arkansas 4 2.4 9 5.4 43 25.7 7 4.2 104 62.3 
Colorado -- -- -- -- 1 10.0 -- -- 9 90.0 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 1 0.8 3 2.3 12 9.2 1 0.8 113 86.9 
Maryland 2 1.5 4 3.1 18 13.8 4 3.1 102 78.5 
Massachusetts 6 1.6 16 4.2 42 11.1 6 1.6 307 81.4 
New Jersey 3 2.1 5 3.5 17 11.8 2 1.4 117 81.3 
New Mexico -- -- 2 5.6 9 25.0 -- -- 25 69.4 
Ohio 2 2.5 1 1.2 12 14.8 -- -- 66 81.5 
Rhode Island -- -- -- -- -- -- -- -- 20 100.0 
All States 21 1.7 41 3.3 202 16.3 30 2.4 947 76.3 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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PARCC Quality of Test Administration 


able H-30. TA Survey (PBA): Q5. The process for identifying accessibility features and accommodations in 


ladvance (via the Personal Needs Profile and associated trainings) was clear and easy to follow. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 2 2.5 5 6.2 38 46.9 5 6.2 31 38.3 
Colorado -- -- 1 8.3 2 16.7 1 8.3 8 66.7 
D.C. -- -- -- -- 2 100.0 -- -- -- -- 
Illinois 17 4.3 33 8.3 84 21.1 12 3.0 253 63.4 
Maryland 2 8.3 2 8.3 6 25.0 2 8.3 12 50.0 
Massachusetts 31 5.4 70 12.1 142 24.6 14 2.4 320 55.5 
Mississippi 1 2.9 3 8.6 7 20.0 4 11.4 20 57.1 
New Jersey - -- 7 28.0 8 32.0 1 4.0 9 36.0 
New Mexico 7 14.6 3 6.3 11 22.9 3 6.3 24 50.0 
Ohio 23 11.2 30 14.6 45 21.8 2 1.0 106 51.5 
Rhode Island 7 5.6 9 7.3 30 24.2 2 1.6 76 61.3 
All States 114 5.5 192 9.2 580 27.8 71 3.4 1,129 54.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-31. TA Survey (EOY): Q5. The process for identifying accessibility features and accommodations in 


ladvance (via the Personal Needs Profile and associated trainings) was clear and easy to follow. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 9 5.0 11 6.1 70 38.7 10 5.5 81 44.8 
Colorado 1 10.0 -- -- 1 10.0 -- -- 8 80.0 
D.C. - -- -- -- 2 100.0 -- -- -- -- 
Illinois 7 5.2 17 12.6 27 20.0 1 0.7 83 61.5 
Maryland 3 2.1 8 5.7 39 27.9 5 3.6 85 60.7 
Massachusetts 26 6.6 40 10.2 118 29.9 10 2.5 200 50.8 
New Jersey 10 6.5 8 5.2 39 25.5 1 0.7 95 62.1 
New Mexico 3 75 6 15.0 16 40.0 1 2.5 14 35.0 
Ohio 3 3.4 10 11.4 21 23.9 -- -- 54 61.4 
Rhode Island -- -- -- -- 8 32.0 -- -- 17 68.0 
All States 72 5.4 107 8.1 408 30.8 40 3.0 698 52.7 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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PARCC Quality of Test Administration 


able H-32. TA Survey (PBA): Q6. Were all the accommodations/accessibility features that were pre-identified 


in the PNP made available to students during a practice session with sample items, tutorial(s), or practice 
est(s)? 


Yes No Not Sure/Don’t Know Not Applicable 
State n % n % n % n % 
Arkansas 21 26.3 9 11.3 30 37.5 20 25.0 
Colorado 1 8.3 -- -- 5 41.7 6 50.0 
D.C. 2 100.0 -- -- -- -- -- -- 
Illinois 74 18.7 25 6.3 154 39.0 142 35.9 
Maryland 6 28.6 -- -- 8 38.1 7 33.3 
Massachusetts 172 30.0 33 5.7 199 34.7 170 29.6 
Mississippi 8 22.9 -- -- 12 34.3 15 42.9 
New Jersey 7 29.2 3 12.5 10 41.7 4 16.7 
New Mexico 13 26.0 7 14.0 15 30.0 15 30.0 
Ohio 46 22.4 12 5.9 78 38.0 69 33.7 
Rhode Island 26 21.1 4 3.3 37 30.1 56 45.5 
All States 567 27.4 108 5.2 716 34.6 679 32.8 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-33. TA Survey (EOY): Q6. Were all the accommodations/accessibility features that were pre-identified in 


he PNP made available to students during a practice session with sample items, tutorial(s), or practice test(s)? 


Yes No Not Sure/Don’t Know Not Applicable 
State n % n % n % n % 
Arkansas 53 29.6 4 2.2 58 32.4 64 35.8 
Colorado 2 18.2 1 9.1 5 45.5 3 27.3 
D.C. 2 100.0 -- -- -- -- -- -- 
Illinois 17 12.7 11 8.2 45 33.6 61 45.5 
Maryland 26 19.0 5 3.6 43 31.4 63 46.0 
Massachusetts 120 30.8 34 8.7 126 32.3 110 28.2 
New Jersey 28 18.3 20 13.1 47 30.7 58 37.9 
New Mexico 12 30.0 4 10.0 10 25.0 14 35.0 
Ohio 17 19.1 5 5.6 29 32.6 38 42.7 
Rhode Island 2 8.0 4 16.8 7 28.0 14 48.0 
All States 339 25.8 90 6.8 430 32.7 455 34.6 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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able H-34. TA Survey (PBA): Q7. During administration, were changes made to a student’s PNP or to the 


availability of accommodations or accessibility features? 


Not Sure/Don’t Know 


Not Applicable 


State n % n % n % n % 

Arkansas 3 3.8 28 35.4 31 39.2 17 21.5 
Colorado -- -- 2 16.7 5 41.7 5 41.7 
D.C. -- -- 1 50.0 1 50.0 -- -- 

Illinois 10 2.5 117 29.4 125 31.4 146 36.7 
Maryland - - 10 45.5 5 22.7 7 31.8 
Massachusetts 30 5.2 230 39.9 169 29.3 147 25.5 
Mississippi - -- 9 25.0 11 30.6 16 44.4 
New Jersey 3 13.6 10 45.5 6 27.3 3 13.6 
New Mexico 3 5.9 19 37.3 12 23.5 17 33.3 
Ohio 4 1.9 67 32.4 70 33.8 66 31.9 
Rhode Island 2 1.6 40 32.3 31 25.0 51 41.1 
All States 66 3.2 765 36.8 595 28.6 655 31.5 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-35. TA Survey (EOY): Q7. During administration, were changes made to a student’s PNP or to the 


availability of accommodations or accessibility features? 


Not Sure/Don’t Know 


Not Applicable 


State n % n % n % n % 

Arkansas 5 2.8 55 31.1 58 32.8 59 33.3 
Colorado 4 9.1 2 18.2 6 54.5 2 18.2 
D.C. 1 50.0 1 50.0 -- -- -- -- 

Illinois 4 2.9 29 21.0 44 31.9 61 44.2 
Maryland 1 0.7 38 27.3 43 30.9 57 41.0 
Massachusetts 23 5.9 157 40.2 116 29.7 95 24.3 
New Jersey 8 5.3 55 36.4 34 22.5 54 35.8 
New Mexico -- -- 21 52.5 6 15.0 13 32.5 
Ohio 2 2.3 29 33.3 20 23.0 36 41.4 
Rhode Island -- -- 9 39.1 6 26.1 8 34.8 
All States 50 3.8 473 35.9 371 28.2 422 32.1 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Sample Items/Tutorials 


Table H-36. TA Survey (PBA): Q1. How did students in your session(s) practice with PARCC content prior to administration? Select all that apply.* 


Sample Items Tutorial Practice Test Did Not Practice Not sure/Don’t Know 

State n % n % n % n % n % 

Arkansas 44 49.4 16 18.0 34 38.2 5 5.6 24 27.0 
Colorado 7 38.9 3 16.7 7 38.9 -- -- 3 16.7 
D.C. -- -- -- -- 1 50.0 1 50.0 -- -- 

Illinois 278 63.2 59 13.4 190 43.2 34 7.7 56 12.7 
Maryland 16 61.5 5 19.2 9 34.6 3 11.5 106 16.0 
Massachusetts 419 63.3 68 10.3 303 45.8 17 2.6 2 7.7 
Mississippi 17 42.5 9 22.5 22 55.0 -- -- 13 32.5 
New Jersey 17 54.8 9 29.0 8 25.8 - -- 7 22.6 
New Mexico 26 45.6 18 31.6 29 50.9 8 14.0 8 14.0 
Ohio 138 57.7 30 12.6 112 46.9 11 4.6 40 16.7 
Rhode Island 84 61.8 10 7.4 50 36.8 10 7.4 21 15.4 
All States” 1,401 60.0 296 12.7 1,083 46.4 99 4.2 435 18.6 


@Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
’N-counts include individuals with missing data for state location who selected a grade level. 
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Sample Items Tutorial Practice Test Did Not Practice Not sure/Don’t Know 
State n % n % n % n % n % 
Arkansas 103 54.5 47 24.9 70 37.0 6 3.2 61 32.3 
Colorado 10 90.9 2 18.2 6 54.5 -- -- 1 9.1 
D.C. 1 50.0 -- -- 1 50.0 -- -- 1 50.0 
Illinois 81 50.6 22 13.8 56 35.0 23 14.4 24 15.0 
Maryland 81 52.6 10 6.5 31 20.1 25 16.2 31 20.1 
Massachusetts 273 60.8 36 8.0 184 41.0 17 3.8 73 16.3 
New Jersey 106 61.3 59 34.1 89 51.4 4 2.3 42 24.3 
New Mexico 30 68.2 11 25.0 22 50.0 2 4.5 3 6.8 
Ohio 53 56.4 12 12.8 44 46.8 10 10.6 21 22.3 
Rhode Island 17 68.0 1 4.0 9 36.0 5 20.0 2 8.0 
All States” 852 58.0 217 14.8 584 39.8 97 6.6 308 21.0 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 


>N-counts include individuals with missing data for state location who selected a grade level. 
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Sample Items Tutorial Practice Test Did Not Practice 

State n % n % n % n % 

Arkansas 50 56.2 30 33.7 29 32.6 23 25.8 
Colorado 8 44.4 3 16.7 7 38.9 2 11.1 
D.C. -- -- -- -- -- -- 2 100.0 
Illinois 329 74.8 180 40.9 233 53.0 40 9.1 
Maryland 18 69.2 14 53.8 14 53.8 1 3.8 
Massachusetts 449 67.8 191 28.9 273 41.2 92 13.9 
Mississippi 19 47.5 16 40.0 15 37.5 6 15.0 
New Jersey 17 54.8 19 61.3 13 41.9 = -- 

New Mexico 37 64.9 21 36.8 31 54.4 9 15.8 
Ohio 158 66.1 91 38.1 111 46.4 34 14.2 
Rhode Island 97 71.3 25 18.4 62 45.6 17 12.5 
All States” 1,585 67.9 744 31.8 1,040 44.5 354 15.2 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
’N-counts include individuals with missing data for state location who selected a grade level. 
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Sample Items Tutorial Practice Test Did Not Practice 
State n % n % n % n % 
Arkansas 118 62.4 77 40.7 77 40.7 42 22.2 
Colorado 8 72.7 5 45.5 10 90.9 -- -- 
D.C. 2 100.0 1 50.0 1 50.0 -- -- 
Illinois 101 63.1 59 36.9 70 43.8 23 14.4 
Maryland 105 68.2 37 24.0 46 29.9 26 16.9 
Massachusetts 306 68.2 129 28.7 199 44.3 65 14.5 
New Jersey 119 68.8 94 54.3 95 54.9 25 14.5 
New Mexico 28 63.6 14 31.8 23 52.3 4 9.1 
Ohio 63 67.0 29 30.9 40 42.6 17 18.1 
Rhode Island 18 72.0 12 48.0 18 72.0 2 8.0 
All States” 976 66.4 503 34.2 646 44.0 247 16.8 


*Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 


>N-counts include individuals with missing data for state location who selected a grade level. 
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able H-40. TA Survey (PBA): Q3. Completion of the PARCC paper-based student tutorials helped better 


prepare me for administering the assessment. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 4 5.1 6 7.6 37 46.8 8 10.1 24 30.4 
Colorado -- -- 1 8.3 3 25.0 -- -- 8 66.7 
D.C. -- -- -- -- 1 50.0 -- -- 1 50.0 
Illinois 21 5.3 55 13.9 182 42.8 33 8.3 106 26.7 
Maryland 1 5.0 2 10.0 9 45.0 3 15.0 5 25.0 
Massachusetts 33 5.7 77 13.2 251 43.1 24 4.1 197 33.8 
Mississippi -- -- 2 5.7 15 42.9 6 17.1 12 34.3 
New Jersey 1 4.3 2 8.7 10 43.5 2 8.7 8 34.8 
New Mexico 4 7.8 6 11.8 20 39.2 4 7.8 17 33.3 
Ohio 18 8.7 33 15.9 86 41.5 5 2.4 65 31.4 
Rhode Island 5 3.9 17 13.4 56 44.1 6 4.7 43 33.9 
All States 107 5.1 253 12.1 904 43.3 128 6.1 698 33.4 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 


able H-41. TA Survey (EOY): Q3. Completion of the PARCC paper-based student tutorials helped better 


prepare me for administering the assessment. 


Strongly Disagree Disagree Agree Strongly Agree Not Applicable 
State n % n % n % n % n % 
Arkansas 7 3.9 11 6.1 77 43.0 14 7.8 70 39.1 
Colorado 4 9.1 -- -- 6 54.5 2 18.2 2 18.2 
D.C. -- -- - -- 2 100.0 -- -- -- -- 
Illinois 10 7.5 21 15.7 58 43.3 6 4.5 39 29.1 
Maryland 3 2.1 21 14.9 55 39.0 5 3.5 57 40.4 
Massachusetts 18 4.6 54 13.7 162 41.0 20 5.1 141 35.7 
New Jersey 8 5.3 8 5.3 46 30.3 5 3.3 85 55.9 
New Mexico 2 5.3 4 10.5 15 39.5 1 2.6 16 42.1 
Ohio 3 3.6 16 19.0 31 36.9 5 6.0 29 34.5 
Rhode Island -- -- 5 20.0 13 52.0 1 4.0 6 24.0 
All States 59 4.5 153 11.6 528 40.1 73 5.5 505 38.3 


Note. Percentages are based on the total number of valid responses. N-counts for “All States” include individuals 
with missing data for state location who selected a response. 
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Appendix | — Item-level Results from Student Survey: 
CBT EOY Mathematics Assessment 


MATH Computer-Based Administration: 


Table I-1. Student Survey Response Rate MATH CBT 
End of Year Assessment 


State n % 

Arkansas 256,924 7.21 
Colorado 354,923 9.96 
District of Columbia 30,933 0.87 
Illinois 703,517 19.74 
Maryland 374,667 10.51 
Massachusetts 127,621 3.58 
Mississippi 210,443 5.91 
New Jersey 732,588 20.56 
New Mexico 187,726 5.27 
Ohio 525,142 14.74 
Rhode Island 59,139 1.66 
All States 3,563,623 100.0 


Table I-2. Student Survey MATH CBT: Q1. Did you understand all of the directions read by the person who 


gave you the test? 


Yes No 

State n % n % 

Arkansas 214,296 83.44 42,525 16.56 
Colorado 289,316 81.60 65,245 18.40 
District of Columbia 25,947 84.01 4,940 15.99 
Illinois 563,374 80.10 139,684 19.90 
Maryland 306,155 81.79 68,166 18.21 
Massachusetts 104,543 81.96 23,007 18.04 
Mississippi 178,441 84.83 31,898 15.17 
New Jersey 587,121 80.20 144,951 19.80 
New Mexico 155,241 82.76 32,342 17.24 
Ohio 419,354 79.90 105,469 20.10 
Rhode Island 48,122 81.43 10,972 18.57 
All States 2,891,910 81.21 669,199 18.79 


Note. Percentages are based on the number of valid responses. Missing = 2,514. 
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Table I-3. Student Survey MATH CBT: Q2. How often was it hard to understand the directions for the questions 


on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arkansas 27,173 10.58 47,950 18.68 117,639 45.82 63,964 24.92 
Colorado 35,980 10.15 69,775 19.69 164,435 46.40 84,202 23.76 
District of Columbia 3,443 11.16 6,689 21.68 12,673 41.08 8,043 26.07 
Illinois 68,797 9.79 137,435 19.55 329,048 46.82 167,560 23.84 
Maryland 41,874 11.20 75,860 20.28 166,791 44.59 89,511 23.93 
Massachusetts 10,757 8.44 20,999 16.47 61,574 48.29 34,172 26.80 
Mississippi 18,174 8.64 38,546 18.33 99,985 47.55 53,575 25.48 
New Jersey 92,280 12.61 147,859 20.21 319,826 43.71 171,761 23.47 
New Mexico 24,991 13.33 42,165 22.48 79,203 42.23 41,172 21.95 
Ohio 55,232 10.53 96,065 18.31 249,176 47.50 124,160 23.67 
Rhode Island 7,380 12.49 11,777 19.94 26,068 44.13 13,845 23.44 
All States 386,081 10.85 695,120 19.53 11626,418 45.69 851,965 23.93 


Note. Percentages are based on the number of valid responses. Missing = 4,039. 


Table I-4. Student Survey MATH CBT: Q3. How many questions asked you about things you have not learned in 


school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arkansas 14,265 5.56 56,255 21.91 126,718 49.39 59,470 23.17 
Colorado 22,213 6.27 79,904 22.57 180,434 50.96 71,486 20.19 
District of Columbia 2,669 8.65 7,259 23.54 13,403 43.46 7,507 24.34 
Illinois 37,314 5.31 162,929 23.19 372,576 53.03 129,768 18.47 
Maryland 25,620 6.85 98,201 26.26 175,206 46.85 74,930 20.04 
Massachusetts 5,568 4,37 20,831 16.34 64,420 50.54 36,642 28.75 
Mississippi 10,352 4.92 30,197 14.36 93,125 44.29 76,576 36.42 
New Jersey 56,741 7.76 191,084 26.13 330,888 45.24 152,665 20.87 
New Mexico 14,945 7.97 49,458 26.38 84,606 45.12 38,497 20.53 
Ohio 27,630 5.27 105,451 20.10 253,596 48.35 137,833 26.28 
Rhode Island 4,119 6.98 14,972 25.36 29,850 50.57 10,091 17.09 
All States 221,436 6.22 816,541 22.95 1,724,822 48.47 795,465 22.36 


Note. Percentages are based on the number of valid responses. Missing = 5,359. 
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Table I-5. Student Survey MATH CBT: Q4. How difficult was this test? 


Easier than School Work Same as School Work Harder than School Work 

State n % n % n % 

Arkansas 19,594 7.63 103,105 40.16 134,009 52.20 
Colorado 34,437 9.73 159,268 45.01 160,116 45.25 
District of Columbia 3,311 10.74 14,689 47.63 12,838 41.63 
Illinois 62,282 8.87 287,263 40.89 352,958 50.24 
Maryland 34,112 9.12 149,110 39.87 190,734 51.00 
Massachusetts 10,258 8.05 56,938 44.68 60,237 47.27 
Mississippi 19,715 9.38 97,591 46.42 92,944 44.21 
New Jersey 58,329 7.98 268,480 36.71 404,446 55.31 
New Mexico 14,477 7.72 75,437 40.23 97,596 52.05 
Ohio 39,816 7.59 213,318 40.67 271,346 51.74 
Rhode Island 4,095 6.94 20,825 35.27 34,117 57.79 
All States 300,426 8.44 1,446,024 40.64 1,811,341 50.91 


Note. Percentages are based on the number of valid responses. Missing = 5,832. 


Table I-6. Student Survey MATH CBT: Q5. Did you have enough time to finish this test? 


Very Early On Time Rush Did Not Finish 
State n % n % n % n % 
Arkansas 116,089 45.23 126,778 49.39 10,719 4.18 3,098 1.21 
Colorado 158,500 44.82 173,644 49.10 16,413 4.64 5,119 1.45 
District of Columbia 10,487 34.04 17,992 58.39 1,621 5.26 711 2.31 
Illinois 308,879 43.98 357,104 50.84 28,378 4.04 7,988 1.14 
Maryland 184,021 49.22 165,931 44.38 18,163 4.86 5,758 1.54 
Massachusetts 52,978 41.61 64,462 50.63 7,657 6.01 2,235 1.76 
Mississippi 73,475 34.95 125,687 59.78 7,949 3.78 3,126 1.49 
New Jersey 363,385 49.72 314,706 43.06 39,832 5.45 12,926 1.77 
New Mexico 62,291 33.22 108,569 57.91 13,657 7.28 2,972 1.59 
Ohio 210,315 40.12 280,884 53.58 26,769 5.11 6,307 1.20 
Rhode Island 25,241 42.76 30,141 51.06 2,763 4.68 887 1.50 
All States 1,565,661 44.02 1,765,898 49.65 173,921 4.89 51,127 1.44 


Note. Percentages are based on the number of valid responses. Missing = 7,016. 
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Table I-7. Student Survey MATH CBT: Q6. How often do you use a computer or tablet at home? 


A Few Times a A Couple Times A Do Not Have at 
Every Day Week Month or Less Home 

State n % n % n % n % 
Arkansas 119,412 46.54 73,205 28.53 35,595 13.87 28,385 11.06 
Colorado 171,888 48.64 111,327 31.50 43,983 12.45 26,211 7.42 
District of Columbia 13,817 44.87 10,437 33.90 4,159 13.51 2,377 7.72 
Illinois 377,890 53.82 209,527 29.84 74,811 10.65 39,930 5.69 
Maryland 194,124 51.96 114,973 30.77 44,282 11.85 20,258 5.42 
Massachusetts 68,934 54.13 40,385 31.71 12,800 10.05 5,232 4.11 
Mississippi 92,724 44.12 62,979 29.97 31,695 15.08 22,763 10.83 
New Jersey 407,502 55.76 215,132 29.44 73,234 10.02 34,891 4.77 
New Mexico 77,188 41.18 59,294 31.63 29,403 15.69 21,550 11.50 
Ohio 284,532 54.29 149,227 28.47 56,252 10.73 34,120 6.51 
Rhode Island 31,854 54.00 18,175 30.81 5,736 9.72 3,220 5.46 
All States 1,839,865 51.75 1,064,661 29.94 411,950 11.59 238,937 6.72 


Note. Percentages are based on the number of valid responses. Missing = 8,210. 


Table I-8. Student Survey MATH CBT: Q7. How often do you use a computer or tablet in school? 


A Couple Times A Do Not Use at 
Every Day A Few Times a Week Month or Less School 

State n % n % n % n % 
Arkansas 78,064 30.42 108,355 42.22 63,421 24.71 6,775 2.64 
Colorado 98,607 27.91 149,658 42.36 93,652 26.51 11,365 3.22 
District of Columbia 8,002 25.99 13,621 44.25 8,073 26.23 1,087 3.53 
Illinois 199,809 28.46 296,956 42.30 186,274 26.53 18,987 2.70 
Maryland 68,632 18.37 143,067 38.29 147,472 39.47 14,422 3.86 
Massachusetts 24,396 19.16 54,877 43.10 44,216 34.73 3,825 3.00 
Mississippi 59,915 28.51 81,307 38.69 59,400 28.27 9,513 4.53 
New Jersey 159,080 21.77 295,916 40.50 247,658 33.90 27,963 3.83 
New Mexico 39,192 20.91 69,083 36.86 69,656 37.19 9,494 5.07 
Ohio 141,402 26.98 227,770 43.46 140,837 26.87 14,074 2.69 
Rhode Island 13,275 22.51 25,211 42.76 18,219 30.90 2,261 3.83 
All States 890,374 25.05 1,465,821 41.23 1,078,878 30.35 119,766 3.37 


Note. Percentages are based on the number of valid responses. Missing = 8,784. 
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able I-9. Student Survey MATH CBT: Q8. When writing a story or essay, how often do you use a computer or 


e]e)(-1irg 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arkansas 29,977 11.68 74,460 29.02 114,129 44.48 38,015 14.82 
Colorado 58,779 16.64 120,292 34.06 133,306 37.75 40,781 11.55 
District of Columbia 5,331 17.33 9,526 30.96 11,593 37.68 4,319 14.04 
Illinois 124,272 17.70 237,910 33.89 256,229 36.50 83,544 11.90 
Maryland 56,304 15.07. 112,444 30.10 152,039 40.70 52,740 14.12 
Massachusetts 19,898 15.62 43,533 34.18 49,065 38.53 14,858 11.67 
Mississippi 22,583 10.75 48,546 23.11 92,367 43.96 46,604 22.18 
New Jersey 123,236 16.87 249,535 34.16 275,032 37.65 82,726 11.32 
New Mexico 24,819 13.25 53,235 28.41 76,901 41.04 32,423 17.30 
Ohio 70,905 13.53 163,136 31.13 214,134 40.87 75,790 14.46 
Rhode Island 10,489 17.79 19,322 32.77 21,673 36.75 7,487 12.70 
All States 546,593 15.38 1,131,939 31.85 1,396,468 39.29 479,287 13.48 


Note. Percentages are based on the number of valid responses. Missing = 9,336. 


Table I-10. Student Survey MATH CBT: Q9. Would you rather take this test on paper OR on a computer or 


tablet? 


On Paper On a Computer or Tablet 
State n % n % 
Arkansas 103,194 40.22 153,371 59.78 
Colorado 113,945 32.27 239,100 67.73 
District of Columbia 6,581 21.39 24,183 78.61 
Illinois 217,155 30.94 484,699 69.06 
Maryland 88,856 23.79 284,614 76.21 
Massachusetts 45,748 35.95 81,523 64.05 
Mississippi 66,289 31.55 143,830 68.45 
New Jersey 265,461 36.36 464,603 63.64 
New Mexico 59,043 31.51 128,333 68.49 
Ohio 198,339 37.85 325,628 62.15 
Rhode Island 16,165 27.43 42,766 72.57 
All States 1,180,776 33.23 2,372,650 66.77 


Note. Percentages are based on the number of valid responses. Missing = 10,197. 
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Table I-11. Student Survey MATH CBT: Q10. What did you use to take this test? 


Desktop Computer Laptop Computer Tablet with a Keyboard Tablet without Keyboard 

State n % n % n % n % 

Arkansas 93,992 36.63 148,075 57.71 12,613 4.92 1,887 0.74 
Colorado 126,328 35.79 206,228 58.43 16,526 4.68 3,895 1.10 
District of Columbia 6,754 21.96 22,816 74.18 882 2.87 304 0.99 
Illinois 271,761 38.72 374,078 53.30 46,750 6.66 9,290 1.32 
Maryland 155,880 41.73 193,714 51.86 20,652 5.53 3,260 0.87 
Massachusetts 38,310 30.09 70,003 54.98 15,310 12.02 3,702 2.91 
Mississippi 138,518 65.93 61,315 29.18 8,057 3.83 2,208 1.05 
New Jersey 221,151 30.28 461,611 63.20 40,877 5.60 6,738 0.92 
New Mexico 117,656 62.78 60,313 32.18 7,183 3.83 2,258 1.20 
Ohio 203,328 38.81 289,516 55.26 27,078 5.17 4,019 0.77 
Rhode Island 16,552 28.08 40,593 68.85 1,365 2.32 445 0.75 
All States 1,390,230 39.12 1,928,262 54.26 197,293 5.55 38,006 1.07 


Note. Percentages are based on the number of valid responses. Missing = 9,832. 
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Table I-12. Student Survey MATH CBT: Q11. Was it easy to type your answers? 


Yes No Did Not Type Answers 
State n % n % n % 
Arkansas 205,919 80.28 36,271 14.14 14,318 5.58 
Colorado 268,179 75.99 51,471 14.59 33,243 9.42 
District of Columbia 23,366 75.99 3,885 12.63 3,499 11.38 
Illinois 550,197 78.41 92,170 13.13 59,366 8.46 
Maryland 289,391 77.50 51,014 13.66 32,991 8.84 
Massachusetts 100,034 78.61 18,126 14.24 9,099 7.15 
Mississippi 172,332 82.04 22,530 10.73 15,191 7.23 
New Jersey 549,217 75.22 122,330 16.75 58,584 8.02 
New Mexico 142,742 76.18 28,284 15.09 16,356 8.73 
Ohio 406,097 77.52 68,588 13.09 49,194 9.39 
Rhode Island 46,856 79.47 8,033 13.63 4,068 6.90 
All States 2,754,330 77.52 502,702 14.15 295,509 8.33 


Note. Percentages are based on the number of valid responses. Missing = 10,682. 


Table I-13. Student Survey MATH CBT: Q12. How many times did you practice on a computer or tablet to get 


ready for this test? 


Never Once More than Once 
State n % n % n % 
Arkansas 59,904 23.35 72,859 28.40 123,747 48.24 
Colorado 122,706 34.78 110,892 31.43 119,223 33.79 
District of Columbia 9,695 31.54 8,512 27.69 12,529 40.76 
Illinois 226,355 32.26 225,252 32.10 250,070 35.64 
Maryland 141,749 37.97 124,038 33.22 107,542 28.81 
Massachusetts 31,876 25.04 36,789 28.90 58,618 46.05 
Mississippi 61,061 29.08 53,928 25.68 95,003 45.24 
New Jersey 186,148 25.50 215,458 29.51 328,525 45.00 
New Mexico 57,196 30.52 57,642 30.76 72,537 38.71 
Ohio 133,234 25.44 150,672 28.77 239,883 45.80 
Rhode Island 17,981 30.51 18,897 32.07 22,055 37.42 
All States 1,047,905 29.50 1,074,939 30.26 1,429,732 40.24 


Note. Percentages are based on the number of valid responses. Missing = 11,047. 


Table I-14. Student Survey MATH CBT: Q13. Was it easy to use the highlighter? 


Yes No Did Not Use Highlighter 

State n % n % n % 

Arkansas 104,951 40.92 11,624 4.53 139,915 54.55 
Colorado 150,745 42.74 20,663 5.86 181,307 51.40 
District of Columbia 13,996 45.56 1,939 6.31 14,787 48.13 
Illinois 295,750 42.16 33,925 4.84 371,844 53.01 
Maryland 155,864 41.76 20,610 5.52 196,773 52.72 
Massachusetts 52,321 41.11 8,508 6.68 66,450 52.21 
Mississippi 100,429 47.82 6,935 3.30 102,631 48.87 
New Jersey 304,261 41.69 48,891 6.70 376,852 51.62 
New Mexico 91,640 48.92 11,070 5.91 84,632 45.18 
Ohio 217,817 41.59 26,917 5.14 279,027 53.27 
Rhode Island 27,889 47.35 3,615 6.14 27,396 46.51 
All States 1,515,663 42.67 194,697 5.48 1,841,614 51.85 


Note. Percentages are based on the number of valid responses. Missing = 11,649. 
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Table I-15. Student Survey MATH CBT: Q14. Was it easy to make pictures or words bigger or smaller? 


Yes No Did Not Change Size 
State n % n % n % 
Arkansas 53,797 20.98 18,244 7.11 184,383 71.91 
Colorado 74,239 21.06 33,531 9.51 244,708 69.43 
District of Columbia 8,477 27.62 2,727 8.89 19,487 63.49 
Illinois 153,181 21.84 52,959 7.55 495,211 70.61 
Maryland 79,914 21.42 30,829 8.26 262,409 70.32 
Massachusetts 28,206 22.17 11,325 8.90 87,715 68.93 
Mississippi 47,266 22.51 11,597 5.52 151,085 71.96 
New Jersey 145,869 19.99 72,818 9.98 511,017 70.03 
New Mexico 49,272 26.30 19,110 10.20 118,931 63.49 
Ohio 103,547 19.77 42,118 8.04 377,962 72.18 
Rhode Island 14,679 24.92 5,664 9.62 38,550 65.46 
All States 758,447 21.36 300,922 8.47 2,491,458 70.17 


Note. Percentages are based on the number of valid responses. Missing = 12,796. 


Table I-16. Student Survey MATH CBT: Q15. Was it easy to enter math symbols and numbers for your answers? 
Did Not Enter Math Symbols 


Yes No or Numbers 

State n % n % n % 

Arkansas 181,537 70.81 45,455 17.73 29,389 11.46 
Colorado 242,017 68.69 71,463 20.28 38,874 11.03 
District of Columbia 21,744 70.86 4,867 15.86 4,074 13.28 
Illinois 493,623 70.40 123,692 17.64 83,811 11.95 
Maryland 265,357 71.15 66,430 17.81 41,188 11.04 
Massachusetts 88,555 69.62 24,792 19.49 13,845 10.89 
Mississippi 161,210 76.81 25,164 11.99 23,514 11.20 
New Jersey 466,231 63.90 162,319 22.25 101,036 13.85 
New Mexico 128,295 68.51 39,037 20.84 19,944 10.65 
Ohio 369,220 70.53 93,616 17.88 60,650 11.59 
Rhode Island 41,710 70.87 10,537 17.90 6,606 11.22 
All States 2,459,499 69.29 667,372 18.80 422,931 11.91 


Note. Percentages are based on the number of valid responses. Missing = 13,821. 


Table I-17. Student Survey MATH CBT: Q16. Was it easy to use the calculator? 


Yes No Did Not Use Calculator 

State n % n % n % 

Arkansas 126,580 49.37 23,448 9.15 106,374 41.49 
Colorado 176,942 50.21 37,005 10.50 138,450 39.29 
District of Columbia 14,758 48.09 2,237 7.29 13,693 44.62 
Illinois 348,908 49.75 61,565 8.78 290,783 41.47 
Maryland 182,750 48.98 33,769 9.05 156,555 41.96 
Massachusetts 53,073 41.72 11,558 9.09 62,585 49.20 
Mississippi 105,624 50.32 10,532 5.02 93,750 44.66 
New Jersey 315,810 43.28 86,833 11.90 327,027 44.82 
New Mexico 99,545 53.15 17,784 9.49 69,976 37.36 
Ohio 228,270 43.60 52,093 9.95 243,197 46.45 
Rhode Island 28,393 48.22 5,290 8.98 25,204 42.80 
All States 1,680,653 47.34 342,114 9.64 1,527,594 43.03 


Note. Percentages are based on the number of valid responses. Missing = 13,262. 
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Table I-18. Student Survey MATH CBT: Q17. Did you have problems logging into the test? 


Yes No 

State n % n % 

Arkansas 56,837 22.17 199,571 77.83 
Colorado 82,205 23.33 270,216 76.67 
District of Columbia 10,357 33.76 20,324 66.24 
Illinois 186,434 26.59 514,800 73.41 
Maryland 89,034 23.87 284,023 76.13 
Massachusetts 35,396 27.82 91,824 72.18 
Mississippi 43,562 20.75 166,358 79.25 
New Jersey 194,869 26.71 534,828 73.29 
New Mexico 49,534 26.44 137,780 73.56 
Ohio 117,094 22.36 406,506 77.64 
Rhode Island 15,769 26.78 43,122 73.22 
All States 881,091 24.82 2,669,352 75.18 


Note. Percentages are based on the number of valid responses. Missing = 13,180. 


Table I-19. Student Survey MATH CBT: Q18. Did the computer (or tablet) stop working? 


Yes No 

State n % n % 

Arkansas 24,698 9.63 231,650 90.37 
Colorado 42,427 12.04 309,823 87.96 
District of Columbia 3,829 12.49 26,828 87.51 
Illinois 82,791 11.81 618,205 88.19 
Maryland 41,265 11.07 331,622 88.93 
Massachusetts 15,784 12.41 111,386 87.59 
Mississippi 15,682 7.47 194,157 92.53 
New Jersey 97,754 13.40 631,655 86.60 
New Mexico 23,789 12.71 163,413 87.29 
Ohio 47,748 9.12 475,676 90.88 
Rhode Island 7,523 12.78 51,330 87.22 
All States 403,290 11.36 3,145,745 88.64 


Note. Percentages are based on the number of valid responses. Missing = 14,588. 


Table I-20. Student Survey MATH CBT: Q19. Did the computer (or tablet) work slowly? 


Yes No 

State n % n % 

Arkansas 56,608 22.08 199,742 77.92 
Colorado 79,141 22.47 273,100 77.53 
District of Columbia 6,467 21.08 24,212 78.92 
Illinois 146,617 20.92 554,371 79.08 
Maryland 71,227 19.10 301,642 80.90 
Massachusetts 28,783 22.64 98,352 77.36 
Mississippi 35,665 17.00 174,169 83.00 
New Jersey 167,827 23.01 561,573 76.99 
New Mexico 41,273 22.04 145,959 77.96 
Ohio 99,265 18.97 424,083 81.03 
Rhode Island 11,280 19.17 47,567 80.83 
All States 744,153 20.97 2,804,770 79.03 


Note. Percentages are based on the number of valid responses. Missing = 14,700. 
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Table I-21. Student Survey MATH CBT: Q20. Did you have a hard time dragging or moving things on the 


screen? 
Yes No 

State n % n % 
Arkansas 65,566 25.58 190,745 74.42 
Colorado 97,243 27.61 254,899 72.39 
District of Columbia 7,113 23.20 23,542 76.80 
Illinois 162,835 23.23 538,004 76.77 
Maryland 89,159 23.92 283,634 76.08 
Massachusetts 32,618 25.66 94,508 74.34 
Mississippi 34,916 16.64 174,871 83.36 
New Jersey 200,799 27.53 528,481 72.47 
New Mexico 43,767 23.38 143,412 76.62 
Ohio 126,418 24.16 396,923 75.84 
Rhode Island 14,309 24.32 44,534 75.68 
All States 874,743 24.65 2,673,553 75.35 


Note. Percentages are based on the number of valid responses. Missing = 15,327. 


Table I-22. Student Survey MATH CBT: Q21. Did you have a hard time making changes to your answers? 


Yes No 

State n % n % 

Arkansas 45,205 17.64 211,070 82.36 
Colorado 64,075 18.20 288,010 81.80 
District of Columbia 5,474 17.85 25,189 82.15 
Illinois 121,681 17.36 579,138 82.64 
Maryland 64,141 17.21 308,646 82.79 
Massachusetts 21,396 16.83 105,730 83.17 
Mississippi 32,269 15.38 177,499 84.62 
New Jersey 151,115 20.72 578,131 79.28 
New Mexico 35,968 19.22 151,205 80.78 
Ohio 85,782 16.39 437,510 83.61 
Rhode Island 10,039 17.07 48,788 82.93 
All States 637,145 17.96 2,910,916 82.04 


Note. Percentages are based on the number of valid responses. Missing = 15,562. 


Table I-23. Student Survey MATH CBT: Q22. Please click on the sentence that is most true: 


No Tech Problems During Test 


Tech Problems During Test 


State n % n % 

Arkansas 203,327 79.33 52,990 20.67 
Colorado 271,236 77.06 80,754 22.94 
District of Columbia 24,115 78.68 6,535 21.32 
Illinois 543,032 77.48 157,813 22.52 
Maryland 191,436 78.43 80,412 21.57 
Massachusetts 95,424 75.06 31,701 24.94 
Mississippi 177,406 84.56 32,392 15.44 
New Jersey 546,107 74.88 183,183 25.12 
New Mexico 144,267 77.06 42,954 22.94 
Ohio 419,511 80.17 103,745 19.83 
Rhode Island 46,126 78.41 12,697 21.59 
All States 2,762,987 77.87 785,176 22.13 


Note. Percentages are based on the number of valid responses. Missing = 15,460. 
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Appendix J — Item-level Results from Student Survey: 
CBT EOY ELA Assessment 


ELA Computer-Based Administration: 


Table J-1. Student Survey Response Rate ELA CBT 
End of Year Assessment 


State n % 

Arkansas 266,298 7.60 
Colorado 381,752 10.90 
District of Columbia 30,799 0.88 
Illinois 713,566 20.37 
Maryland 335,775 9.58 
Massachusetts 127,410 3.64 
Mississippi 202,001 5.77 
New Jersey 755,599 21.57 
New Mexico 188,638 5.38 
Ohio 443,902 12.67 
Rhode Island 57,734 1.65 
All States 3,503,474 100.0 


Table J-2. Student Survey ELA CBT: Q1. Did you understand all of the directions read by the person who gave 


you the test? 


Yes No 

State n % n % 

Arkansas 235,505 88.47 60,699 11.53 
Colorado 333,123 87.33 48,346 12.67 
District of Columbia 27,300 88.74 3,463 11.26 
Illinois 619,895 86.91 93,331 13.09 
Maryland 294,312 87.70 41,279 12.30 
Massachusetts 111,612 87.64 15,746 12.36 
Mississippi 182,126 90.19 19,812 9.81 
New Jersey 650,149 86.09 105,059 13.91 
New Mexico 167,474 88.80 21,075 11.20 
Ohio 388,320 87.52 55,396 12.48 
Rhode Island 50,045 86.72 7,663 13.28 
All States 3,059,861 87.38 441,869 12.62 


Note. Percentages are based on the number of valid responses. Missing = 1,744. 
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Table J-3. Student Survey ELA CBT: Q2. How often was it hard to understand the directions for the questions 


on this test? 


Almost Always 


Most of the Time 


Some of the Time 


Almost Never 


State n % n % n % n % 

Arkansas 17,638 6.63 36,713 13.80 123,544 46.43 88,205 33.15 
Colorado 26,562 6.97 55,205 14.48 181,961 47.73 117,468 30.82 
District of Columbia 2,426 7.90 5,381 17.52 13,558 44.14 9,354 30.45 
Illinois 47,386 6.65 103,821 14.56 344,534 48.33 217,210 30.47 
Maryland 23,610 7.04 49,394 14.73 157,250 46.89 105,098 31.34 
Massachusetts 7,009 5.50 16,661 13.09 64,074 50.34 39,536 31.06 
Mississippi 10,795 5.35 28,307 14.02 102,422 50.74 60,328 29.89 
New Jersey 64,397 8.53 119,100 15.78 353,959 46.90 217,285 28.79 
New Mexico 14,899 7.90 32,757 17.38 90,206 47.86 50,621 26.86 
Ohio 24,554 5.54 58,303 13.14 232,668 52.42 128,088 28.87 
Rhode Island 5,019 8.70 8,878 15.40 25,318 43.91 18,449 31.99 
All States 244,295 6.98 514,520 14.70 1,689,494 48.27 1,051,64 30.05 

2 


Note. Percentages are based on the number of valid responses. Missing = 3,523. 


Table J-4. Student Survey ELA CBT: Q3. How many questions asked you about things you have not learned in 


school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arkansas 13,709 5.15 39,826 14.97 127,302 47.85 85,209 32.03 
Colorado 20,924 5.50 63,128 16.58 193,185 50.75 103,427 27.17 
District of Columbia 2,298 7.49 5,480 17.86 13,508 44.03 9,394 30.62 
Illinois 36,237 5.09 121,402 17.04 367,506 51.58 187,404 26.30 
Maryland 20,335 6.07 59,163 17.66 161,957 48.34 93,599 27.94 
Massachusetts 5,115 4.02 17,192 13.52 63,621 50.01 41,276 32.45 
Mississippi 9,313 4.61 20,904 10.36 83,315 41.28 88,278 43.74 
New Jersey 56,087 7.44 149,131 19.78 341,236 45.25 207,642 27.54 
New Mexico 11,383 6.04 36,166 19.19 93,199 49.46 47,667 25.30 
Ohio 15,925 3.59 64,344 14.51 229,705 51.81 133,427 30.09 
Rhode Island 4,068 7.06 10,440 18.12 26,891 46.67 16,223 28.15 
All States 195,394 5.59 587,176 16.79 1,701,425 48.65 1,013,546 28.98 


Note. Percentages are based on the number of valid responses. Missing = 5,933. 
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Table J-5. Student Survey ELA CBT: Q4. How difficult was this test? 


Easier than School Work Same as School Work Harder than School Work 

State n % n % n % 

Arkansas 30,343 11.40 135,698 51.00 100,032 37.60 
Colorado 56,248 14.78 209,149 54.97 115,090 30.25 
District of Columbia 3,706 12.08 18,134 59.10 8,842 28.82 
Illinois 85,200 11.96 362,251 50.84 265,063 37.20 
Maryland 44,290 13.21 177,698 53.02 113,182 33.77 
Massachusetts 12,818 10.08 66,028 51.91 48,359 38.02 
Mississippi 19,728 9.78 113,523 56.25 68,558 33.97 
New Jersey 71,599 9.49 342,912 45.47 339,582 45.03 
New Mexico 19,640 10.42 96,242 51.07 72,586 38.51 
Ohio 39,026 8.80 221,191 49.88 183,199 41.32 
Rhode Island 5,778 10.03 26,291 45.63 25,550 44.34 
All States 388,376 11.10 1,769,117 50.58 1,340,043 38.31 


Note. Percentages are based on the number of valid responses. Missing = 5,938. 


Table J-6. Student Survey ELA CBT: Q5. Did you have enough time to finish this test? 


Very Early On Time Rush Did Not Finish 
State n % n % n % n % 
Arkansas 128,580 48.34 124,960 46.97 9,351 3.52 3,125 1.17 
Colorado 179,952 47.31 179,985 47.32 14,981 3.94 5,436 1.43 
District of Columbia 10,121 33.02 17,835 58.19 1,682 5.49 1,009 3.29 
Illinois 323,149 45.36 353,462 49.62 26,883 3.77 8,842 1.24 
Maryland 172,873 51.59 143,896 42.95 13,547 4.04 4,746 1.42 
Massachusetts 52,169 41.03 64,859 51.02 7,730 6.08 2,376 1.87 
Mississippi 68,391 33.90 117,670 58.33 10,248 5.08 5,407 2.68 
New Jersey 373,390 49.54 327,730 43.48 40,265 5.34 12,367 1.64 
New Mexico 63,395 33.65 108,047 57.34 13,689 7.27 3,290 1.75 
Ohio 185,342 41.81 232,799 52.52 20,439 4.61 4,681 1.06 
Rhode Island 26,214 45.51 28,206 48.96 2,354 4.09 831 1.44 
All States 1,583,576 45.29 1,699,449 48.61 161,169 4.61 52,110 1.49 


Note. Percentages are based on the number of valid responses. Missing = 7,170. 
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Table J-7. Student Survey ELA CBT: Q6. How often do you use a computer or tablet at home? 


A Few Times a A Couple Times A Do Not Have at 
Every Day Week Month or Less Home 

State n % n % n % n % 
Arkansas 125,415 47.16 74,585 28.05 36,766 13.83 29,164 10.97 
Colorado 188,248 49.54 118,097 31.08 46,045 12.12 27,594 7.26 
District of Columbia 14,084 45.97 10,232 33.40 3,914 12.78 2,407 7.86 
Illinois 389,652 54.72 208,631 29.30 73,420 10.31 40,370 5.67 
Maryland 176,399 52.67. 102,395 30.58 38,435 11.48 17,662 5.27 
Massachusetts 69,996 55.06 39,843 31.34 12,004 9.44 5,291 4.16 
Mississippi 89,989 44.62 59,937 29.72 30,103 14.93 21,651 10.74 
New Jersey 431,569 57.27. 215,357 28.58 71,627 9.50 35,023 4.65 
New Mexico 79,979 42.46 58,694 31.16 28,392 15.07 21,309 11.31 
Ohio 257,401 58.08 117,259 26.46 42,739 9.64 25,797 5.82 
Rhode Island 31,468 54.66 17,585 30.54 5,534 9.61 2,988 5.19 
All States 1,854,200 53.05 1,022,615 29.26 388,979 11.13 229,256 6.56 


Note. Percentages are based on the number of valid responses. Missing = 8,424. 


Table J-8. Student Survey ELA CBT: Q7. How often do you use a computer or tablet in school? 


A Couple Times A Do Not Use at 
Every Day A Few Times a Week Month or Less School 

State n % n % n % n % 
Arkansas 82,196 30.91 111,538 41.94 65,012 24.45 7,178 2.70 
Colorado 104,966 27.63 159,306 41.94 102,934 27.10 12,628 3.32 
District of Columbia 8,018 26.19 13,372 43.68 8,083 26.40 1,140 3.72 
Illinois 207,305 19.12 298,913 41.99 185,957 = 26.12 19,751 2.77 
Maryland 63,731 19.03 129,409 38.65 128,378 38.34 13,2983 3.97 
Massachusetts 25,148 19.79 54,865 43.17 43,163 33.96 3,910 3.08 
Mississippi 56,273 27.91 77,596 38.48 57,547 28.54 10,226 5.07 
New Jersey 170,505 22.63 302,399 40.14 252,073 33.46 28,325 3.76 
New Mexico 40,141 21.31 68,691 36.47 70,092 37.21 9,440 5.01 
Ohio 126,236 28.49 190,780 43.06 116,587 26.31 9,476 2.14 
Rhode Island 13,130 22.81 24,623 42.78 17,687 30.73 2,115 3.67 
All States 897,649 25.69 1,431,492 40.97 1,047,513 29.98 117,487 3.36 


Note. Percentages are based on the number of valid responses. Missing = 9,333. 
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able J-9. Student Survey ELA CBT: Q8. When writing a story or essay, how often do you use a computer or 


e]e)(-1irg 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arkansas 30,367 11.42 76,930 28.93 119,539 44.96 39,070 14.69 
Colorado 61,138 16.10 128,718 33.89 146,040 38.45 43,883 11.55 
District of Columbia 5,179 16.91 9,640 31.47 11,837 38.64 3,978 12.99 
Illinois 123,966 17.41 270,825 33.83 264,057 37.09 83,012 11.66 
Maryland 46,868 14.00 99,071 29.59 141,368 42.22 47,546 14.20 
Massachusetts 18,879 14.85 42,586 33.50 50,745 39.92 14,907 11.73 
Mississippi 19,691 9.77 45,242 22.44 91,754 45.52 44,903 22.27 
New Jersey 124,825 16.57 256,826 34.09 287,091 38.11 84,569 11.23 
New Mexico 23,747 12.61 52,977 28.13 79,348 42.13 32,280 17.14 
Ohio 58,895 13.29 146,176 32.99 184,859 41.72 53,157 12.00 
Rhode Island 10,091 17.52 18,779 32.61 21,641 37.58 7,080 12.29 
All States 523,646 14.99 1,117,770 31.99 1,398,279 40.02 454,385 13.00 


Note. Percentages are based on the number of valid responses. Missing = 9,394. 


Table J-10. Student Survey ELA CBT: Q9. Would you rather take this test on paper OR on a computer or tablet? 


On Paper On a Computer or Tablet 
State n % n % 
Arkansas 101,474 38.16 164,431 61.84 
Colorado 114,693 30.22 264,825 69.78 
District of Columbia 6,316 20.62 24,315 79.38 
Illinois 210,536 29.58 501,193 70.42 
Maryland 70,325 21.00 264,494 79.00 
Massachusetts 43,465 34.21 83,597 65.79 
Mississippi 58,958 29.25 142,641 70.75 
New Jersey 263,472 35.00 489,338 65.00 
New Mexico 56,889 30.21 131,438 69.79 
Ohio 169,600 38.28 273,416 61.72 
Rhode Island 14,662 25.47 42,900 74.53 
All States 1,110,390 31.79 2,382,588 68.21 


Note. Percentages are based on the number of valid responses. Missing = 10,496. 
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Table J-11. Student Survey ELA CBT: Q10. What did you use to take this test? 


Desktop Computer Laptop Computer Tablet with a Keyboard Tablet without Keyboard 

State n % n % n % n % 

Arkansas 96,603 36.33 154,573 58.14 12,787 4.81 1,920 0.72 
Colorado 138,951 36.61 218,121 57.48 18,225 4.80 4,199 1.11 
District of Columbia 6,588 21.53 22,858 74.69 861 2.81 298 0.97 
Illinois 276,510 38.85 377,647 53.06 48,665 6.84 8,877 1.25 
Maryland 129,692 38.73 183,990 54.95 18,079 5.40 3,064 0.92 
Massachusetts 37,730 29.69 70,422 55.41 15,430 12.14 3,519 2.77 
Mississippi 131,439 65.22 59,197 29.37 8,117 4.03 2,792 1.39 
New Jersey 225,821 29.98 477,972 63.46 42,994 5.71 6,420 0.85 
New Mexico 117,394 62.32 60,859 32.31 7,920 4.20 2,193 1.16 
Ohio 162,670 36.72 256,616 57.92 20,557 4.64 3,183 0.72 
Rhode Island 16,504 28.66 39,472 68.55 1,209 2.10 393 0.68 
All States 1,339,902 38.36 1,921,727 55.01 194,844 5.58 36,858 1.06 


Note. Percentages are based on the number of valid responses. Missing = 10,143. 
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Table J-12. Student Survey ELA CBT: Q11. Was it easy to type your answers? 


Yes No Did Not Type Answers 
State n % n % n % 
Arkansas 136,721 51.43 18,783 7.07 110,346 41.51 
Colorado 207,058 54.58 34,485 9.09 137,817 36.33 
District of Columbia 15,152 49.52 2,417 7.90 13,026 42.58 
Illinois 376,205 52.87 55,025 7.73 280,360 39.40 
Maryland 172,657 51.58 23,841 7.12 138,256 41.30 
Massachusetts 71,016 55.89 11,873 9.34 44,164 34.76 
Mississippi 81,452 40.42 8,954 4.44 111,093 55.13 
New Jersey 411,180 54.61 82,764 10.99 259,019 34.40 
New Mexico 109,563 58.18 18,264 9.70 60,485 32.12 
Ohio 209,088 47.20 31,962 7.22 201,932 45.58 
Rhode Island 33,728 58.58 4,712 8.18 19,136 33.24 
All States 1,823,820 52.22 293,080 8.39 1,375,634 39.39 


Note. Percentages are based on the number of valid responses. Missing = 10,940. 


Table J-13. Student Survey ELA CBT: Q12. How many times did you practice on a computer or tablet to get ready 


for this test? 


Never Once More than Once 
State n % n % n % 
Arkansas 66,300 24.94 74,077 27.87 125,448 47.19 
Colorado 138,715 36.58 114,868 30.29 125,655 33.13 
District of Columbia 9,985 32.64 7,954 26.00 12,651 41.36 
Illinois 238,019 33.46 220,610 31.01 252,816 35.54 
Maryland 123,960 37.04 107,120 32.01 103,613 30.96 
Massachusetts 32,048 25.23 34,399 27.08 60,601 47.70 
Mississippi 58,960 29.27 48,044 23.85 94,414 46.87 
New Jersey 201,472 26.76 214,237 28.46 337,177 44.78 
New Mexico 58,970 31.32 56,021 29.75 73,299 38.93 
Ohio 108,770 24.56 122,839 27.73 211,309 47.71 
Rhode Island 18,118 31.48 18,153 31.54 21,285 36.98 
All States 1,055,317 30.22 1,018,322 29.16 1,418,268 40.62 


Note. Percentages are based on the number of valid responses. Missing = 11,567. 


Table J-14. Student Survey ELA CBT: Q13. Was it easy to use the highlighter? 


Yes No Did Not Use Highlighter 

State n % n % n % 

Arkansas 135,358 50.92 13,532 5.09 116,912 43.98 
Colorado 192,767 50.85 23,017 6.07 163,322 43.08 
District of Columbia 17,929 58.68 2,069 6.77 10,557 34.55 
Illinois 374,686 52.67 36,322 5.11 300,330 42.22 
Maryland 172,195 51.46 19,838 5.93 142,588 42.61 
Massachusetts 65,856 51.84 8,748 6.86 52,429 41.27 
Mississippi 127,926 63.53 7,323 3.64 66,118 32.83 
New Jersey 382,012 50.75 51,754 6.88 318,998 42.38 
New Mexico 111,952 59.46 10,491 5.57 65,836 34.97 
Ohio 242,449 54.74 21,518 4.86 178,921 40.40 
Rhode Island 31,632 54.96 3,715 6.45 22,209 38.59 
All States 1,854,762 53.13 198,327 5.68 1,438,220 41.19 


Note. Percentages are based on the number of valid responses. Missing = 12,165. 


Submitted September 28, 2015 Page J-7 


PARCC Quality of Test Administration 


Table J-15. Student Survey ELA CBT: Q14. Was it easy to make pictures or words bigger or smaller? 


Yes No Did Not Change Size 
State n % n % n % 
Arkansas 54,951 20.68 18,219 6.86 162,582 72.47 
Colorado 78,206 20.64 35,177 9.29 265,465 70.07 
District of Columbia 8,177 26.76 2,826 9.26 19,525 63.96 
Illinois 156,012 21.94 54,128 7.61 500,941 70.45 
Maryland 70,521 21.08 27,137 8.11 236,823 70.80 
Massachusetts 27,617 21.75 11,238 8.85 88,098 69.39 
Mississippi 46,251 22.98 11,470 5.70 143,585 71.33 
New Jersey 146,586 19.48 74,438 9.89 531,437 70.63 
New Mexico 47,705 25.35 18,404 9.78 122,088 64.87 
Ohio 85,738 19.37 34,423 7.77 322,582 72.86 
Rhode Island 13,719 23.85 5,067 8.81 38,726 67.34 
All States 735,483 21.07 292,527 8.38 2,461,852 70.54 


Note. Percentages are based on the number of valid responses. Missing = 13,612. 


Table J-16. Student Survey ELA CBT: Q15. Was it easy to move back and forth between passages or stories? 


Did Not Move 
Yes No Between Passages or Stories 

State n % n % n % 

Arkansas 207,727 78.21 22,346 8.41 35,538 13.38 
Colorado 293,718 77.58 35,648 9.42 49,233 13.00 
District of Columbia 24,058 78.89 3,121 10.23 3,316 10.87 
Illinois 555,086 78.10 59,529 8.38 96,156 13.53 
Maryland 266,801 79.80 28,941 8.66 38,534 11.53 
Massachusetts 101,835 80.26 11,762 9.27 13,286 10.47 
Mississippi 167,994 83.50 12,830 6.38 20,363 10.12 
New Jersey 566,608 75.35 88,868 11.82 96,536 12.84 
New Mexico 150,730 80.12 17,884 9.51 19,509 10.37 
Ohio 357,266 80.73 35,273 7.97 49,985 11.30 
Rhode Island 44,760 77.88 5,335 9.28 7,378 12.84 
All States 2,736,583 78.46 321,537 9.22 429,834 12.32 


Note. Percentages are based on the number of valid responses. Missing = 15,520. 


Table J-17. Student Survey ELA CBT: Q16. Was it easy to find information in the passages or stories when 


answering questions? 


Did Not Move 
Yes No Between Passages or Stories 

State n % n % n % 

Arkansas 193,695 72.93 57,006 21.46 14,896 5.61 
Colorado 274,885 72.65 78,964 20.87 24,529 6.48 
District of Columbia 23,043 75.58 5,789 18.99 1,657 5.43 
Illinois 517,032 72.76 149,227 21.00 44,292 6.23 
Maryland 246,710 73.82 69,693 20.82 17,800 5.33 
Massachusetts 92,679 73.07 28,173 22.21 5,977 4.71 
Mississippi 159,674 79.40 34,439 17.12 6,993 3.48 
New Jersey 490,007 65.17 212,865 28.31 48,961 6.51 
New Mexico 136,065 72.34 42,222 22.45 9,807 5.21 
Ohio 327,201 73.94 96,184 21.74 19,110 4.32 
Rhode Island 41,630 72.44 12,192 21.22 3,644 6.34 
All States 2,502,621 71.77 786,754 22.56 197,666 5.67 


Note. Percentages are based on the number of valid responses. Missing = 16,433. 
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Table I-18. Student Survey ELA CBT: Q17. Did you have problems logging into the test? 


Yes No 

State n % n % 

Arkansas 55,430 20.86 210,254 79.14 
Colorado 84,461 22.31 294,178 77.69 
District of Columbia 10,032 32.88 20,482 67.12 
Illinois 184,989 26.02 525,915 73.98 
Maryland 75,722 22.65 258,665 77.35 
Massachusetts 33,858 26.67 93,074 73.33 
Mississippi 43,444 21.59 157,735 78.41 
New Jersey 195,327 25.96 557,014 74.04 
New Mexico 50,421 26.79 167,784 73.21 
Ohio 96,988 21.91 345,765 78.09 
Rhode Island 14,233 24.76 43,254 75.24 
All States 844,905 24.22 2,644,120 75.78 


Note. Percentages are based on the number of valid responses. Missing = 14,449. 


Table I-19. Student Survey ELA CBT: Q18. Did the computer (or tablet) stop working? 


Yes No 

State n % n % 

Arkansas 22,184 8.35 243,362 91.65 
Colorado 41,533 10.98 336,848 89.02 
District of Columbia 3,438 11.28 27,044 88.72 
Illinois 76,614 10.78 634,007 89.22 
Maryland 33,277 9.96 300,862 90.04 
Massachusetts 14,429 11.37 112,424 88.63 
Mississippi 13,447 6.69 187,606 93.31 
New Jersey 91,516 12.17 660,406 87.83 
New Mexico 21,593 11.48 166,487 88.52 
Ohio 36,544 8.26 406,042 91.74 
Rhode Island 6,354 11.06 51,112 88.94 
All States 360,929 10.35 3,126,200 89.65 


Note. Percentages are based on the number of valid responses. Missing = 16,345. 


Table I-20. Student Survey ELA CBT: Q19. Did the computer (or tablet) work slowly? 


Yes No 

State n % n % 

Arkansas 50,833 19.14 214,717 80.86 
Colorado 76,613 20.25 301,737 79.75 
District of Columbia 6,053 19.85 24,434 80.15 
Illinois 135,682 19.10 574,878 80.90 
Maryland 54,964 16.45 279,175 83.55 
Massachusetts 25,356 19.99 101,502 80.01 
Mississippi 31,548 15.69 169,516 84.31 
New Jersey 155,626 20.70 596,192 79.30 
New Mexico 38,099 20.25 150,002 79.75 
Ohio 72,422 16.37 370,099 83.63 
Rhode Island 9,596 16.70 47,860 83.30 
All States 656,792 18.84 2,830,112 81.16 


Note. Percentages are based on the number of valid responses. Missing = 16,570. 
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Table I-21. Student Survey ELA CBT: Q20. Did you have a hard time dragging or moving things on the screen? 


Yes No 

State n % n % 

Arkansas 72,282 27.22 193,263 72.78 
Colorado 112,992 29.87 265,342 70.13 
District of Columbia 8,841 29.01 21,633 70.99 
Illinois 174,550 24.57 536,009 75.43 
Maryland 87,349 26.15 246,706 73.82 
Massachusetts 35,494 27.98 91,342 72.02 
Mississippi 36,553 19.68 161,440 80.32 
New Jersey 226,150 30.08 525,637 69.92 
New Mexico 51,711 27.50 136,347 72.50 
Ohio 115,382 26.08 327,101 73.92 
Rhode Island 14,243 24.79 43,212 75.21 
All States 938,547 26.92 2,548,032 73.08 


Note. Percentages are based on the number of valid responses. Missing = 16,895. 


Table I-22. Student Survey ELA CBT: Q21. Did you have a hard time making changes to your answers? 


Yes No 

State n % n % 

Arkansas 10,918 15.41 224,594 84.59 
Colorado 61,658 16.30 316,530 83.70 
District of Columbia 5,320 17.45 25,159 82.55 
Illinois 112,727 15.87 597,638 84.13 
Maryland 51,816 15.51 282,272 84.49 
Massachusetts 19,615 15.47 107,217 84.53 
Mississippi 30,807 15.34 170,085 84.66 
New Jersey 142,088 18.90 609,563 81.10 
New Mexico 32,528 17.30 155,494 82.70 
Ohio 60,883 13.76 381,532 86.24 
Rhode Island 8,668 15.09 48,789 84.91 
All States 567,029 16.27 2,918,873 83.73 


Note. Percentages are based on the number of valid responses. Missing = 17,572. 


Table I-23. Student Survey ELA CBT: Q22. Please click on the sentence that is most true: 


No Tech Problems During Test 


Tech Problems During Test 


State n % n % 

Arkansas 216,227 81.44 49,266 18.56 
Colorado 297,997 78.82 80,068 21.18 
District of Columbia 24,319 79.82 6,147 20.18 
Illinois 560,842 78.94 149,604 21.06 
Maryland 267,790 80.15 66,338 19.85 
Massachusetts 96,810 76.33 30,024 23.67 
Mississippi 171,966 85.58 28,966 14.42 
New Jersey 573,119 76.23 178,683 23.77 
New Mexico 176,751 78.00 41,384 22.00 
Ohio 361,001 81.60 81,419 18.40 
Rhode Island 46,206 80.42 11,253 19.58 
All States 2,763,028 79.26 723,152 20.74 


Note. Percentages are based on the number of valid responses. Missing = 17,294. 
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Appendix K — Specific Suggestions Obtained from TAs during Debriefing Interviews for 
Improving TA Manual, Scripts and Training 


While conducting debriefing interviews, TAs sometimes provided suggestions for improvement 
to PARCC processes and procedures. Those recommendations were documented and are 
presented by topic below. 


Manuals 


Verify consistent use of terminology in all written materials. 
Provide checklists for common processes such as starting and ending sessions, and 
possibly include screenshots for each step. 

3. Provide additional guidance on managing exceptions and responding to student 
questions. 
Include a trouble-shooting guide for common technology issues. 

5. Add an index to the manual. 


Scripts 


1. Adda reminder to the TA script to start the session on the TA computer prior to starting 
the script. 

2. Streamline the scripts to minimize redundancy between sessions. 

3. Include the option for TAs to provide instructions specific to the accommodation or 
accessibility feature students may access. 
Eliminate the universal directions to test headphones. 

5. Include all of the directions as text on the screen for students to follow along as the TA 
reads them the directions from the script. 

6. Clarify instructions for navigating between sections and sessions. 
Provide instructions in the script for the Student Survey. 


Training 


1. Revisit all training materials to verify consistency of language and format with the TA 
and TC test interface and the student view of the assessment. 

2. Develop additional training materials that provide detailed instructions on common 
tasks, such as skipping sessions for a student who was absent, setting up student 
computers, and starting or resuming test sessions. 

3. Develop a best practices guideline document based on successful experiences of school 
and districts, such as providing drop-in labs for TCs and TAs to learn from each other. 
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